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ABSTRACT 


Top-level  acquisition  decisions  are  complex  and  multi¬ 
objective.  This  implies  difficulties  in  appropriately 
accounting  for  all  relevant  factors  to  select  the  best 
alternative.  In  addition,  there  are  difficulties  and  defi¬ 
ciencies  in  the  actual  implementation  of  the  decision  proc¬ 
ess.  The  difficulties  and  deficiencies  in  the  top-level 
acquisition  decision-making  are  explored  by  this  thesis. 

The  thesis  suggests  improvements  through  intensive  use  of 
a  quantitative,  judgment-based  decision  technique  derived 
from  Multi-Attribute  Utility  Theory  (MAUT) .  Emphasis  is 
put  on  demonstrating  MAUT's  capability  for  incorporating 
subjective  judgment,  in  order  to  reduce  the  existing  doubts 
about  its  usefulness  for  top-level  decision-making.  The 
thesis  recommends  use  of  the  MAUT  procedure  as  the  central 
tool  for  comprehensive  evaluation  of  the  decision  alternatives 
It  argues  that  such  use  would  solve  some  of  the  essential 
decision-making  problems  and  in  addition  contribute  to  the 
quality  and  efficiency  of  the  decision  process. 
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I.  INTRODUCTION 


A .  GENERAL 

The  acquisition  of  major  systems  by  the  Federal 
Government  constitutes  one  of  the  most  crucial 
and  expensive  activities  performed  to  meet 
national  needs.  Its  impact  is  critical  on  tech¬ 
nology,  on  the  nation's  economic  and  fiscal 
policies,  and  on  the  accomplishment  of  Govern¬ 
ment  agency  missions  in  such  fields  as  defense, 
space,  energy  and  transportation.  [Circular  No. 

A-109 ,  p.  1] 

The  above  quote  clearly  indicates  how  crucial,  and  complex 
the  major  systems  acquisition  decisions  are.  The  quality  of 
these  decisions  have  long-run  implications  not  only  for  the 
national  defense  capabilities,  but  for  political  and  socio¬ 
economic  issues  as  well. 

The  complexity  of  the  major  systems  acquisition  decisions 
stems  from  the  scope  and  substance  of  factors  such  as: 

(1)  The  variety  of  organizations  and  personalities  with 
interests  in  the  decision. 

(2)  The  enormous  resources  committed  to  the  decision. 

(3)  The  technological  and  financial  uncertainties  involved 
in  programs  whose  acquisition  life  cycle  stretches  over  7,  10, 
15  or  more  years,  and  which  exploit  the  leading  edge  of  the 
technological  state  of  the  art. 

(4)  The  large  and  complex  organizations  and  management 
involved  in  running  the  programs  through  their  acquisition 
life  cycle. 

During  the  acquisition  life  cycle  there  is  a  continuous 
decision-making  process,  involving  various  echelons,  up  to 
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the  top  decision-maker ,  which,  for  major  systems,  is  the 
Secretary  of  Defense,  or  in  some  cases,  even  the  President 
himself . 

The  decision-making  process  works  in  a  hierarchical  manner. 

As  a  decision  issue  ascends  the  decision-making  ladder,  it 
becomes  more  refined,  the  alternatives  are  better  defined, 
and  their  number  is  reduced.  At  the  same  time  another  develop¬ 
ment  takes  place:  As  the  decision  moves  up,  emphasis  changes, 
new  considerations  are  added,  and  in  addition  to  the  measurable 
factors  more  and  more  intangible  factors,  which  are  subject 
to  judgment,  become  involved. 

Thus,  top-level  acquisition  decision-making  is  distin¬ 
guished  by  the  problem  of  facing  a  broad  spectrum  of  decision 
factors,  of  various  scopes  and  natures,  some  of  them  technically 
measurable,  and  some  of  them  more  judgmental.  Top-level 
decision-makers  have  the  difficult  task  of  intelligently  con¬ 
solidating  all  these  factors  and  considerations  into  one  final 
decision.  The  difficulty  of  such  consolidation  is  amplified 
by  the  previously  mentioned  substantial  implications  of  the 
acquisition  decisions. 

Because  of  their  complexity  and  crucial  implications,  major 
systems  acquisition  decisions  have  always  been  subject  to  con¬ 
troversy,  criticism  and  concern.  Circular  No.  A- 109  [Execu¬ 
tive  Office  of  the  President,  1976]  stated  several  years  ago 
that  "for  a  number  of  years  there  has  been  deep  concern  over 
the  effectiveness  of  the  management  of  major  system  acquisition." 
This  thesis  reveals  that  concerns  and  criticism  about  these 
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issues  have  continued  to  the  present.  In  fact,  there  always 
should  be  concern  about  acquisition  decisions,  and  there  are 
no  limits  to  striving  for  their  improvement. 

B.  OBJECTIVES 

The  objectives  of  this  study  were  to  identify  problems  and 
deficiencies  in  the  current  methodology  by  which  acquisition 
decision  alternatives  are  evaluated,  compared  and  presented 
for  top-level  decision-making,  and  to  suggest  use  of  a  decision 
technique  that  should  improve  the  way  the  ultimate  decision 
is  made. 

C.  SCOPE 

The  scope  of  this  thesis  is  limited  to  the  search  for  an 
evaluation  framework  or  decision-making  technique  that  can 
best  serve  top-level  acquisition  decision-making.  The  thesis 
does  not  directly  address  the  major  problems  of  acquisition 
decisions  such  as  how  to  translate  threat  assessment  into 
weapon  systems  terms,  how  to  reliably  predict  cost,  performance 
and  schedule,  or  how  to  measure  the  political  and  socio-economic 
implications  of  the  decision.  These  are  beyond  the  scope  of 
the  thesis.  Rather,  the  thesis  concentrates  on  the  question 
of  how  to  best  consolidate  the  above  inputs  to  an  ultimate 
decision,  under  the  assumption  that  they  are  already  given. 

The  thesis  does  not  attempt  to  recommend  specific  changes 
in  the  formal  acquisition  decision  process,  the  so-called 
"DSARC  Process"  (which  is  actually  in  the  process  of  change) . 
Rather,  it  does  address  the  underlying  approaches  to 
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decision-making  which  are  not  necessarily  attached  to  one 
formal  process  or  another. 

The  study  refers  mostly  to  top-level  decision-making. 

In  this  context,  top-level  acquisition  decision-makers  are 
defined  as  the  highest  acquisition  authorities,  such  as  Source 
Selection  Board,  (Service) SARC,  (Service) Sec — at  the  Service 
Department  level;  DSARC  and  SecDef — at  the  OSD  level.1  But 
this  definition  is  by  no  means  strict.  Any  decision  body 
that  meets  the  thesis'  principal  assumptions  may  be  considered 
as  addressed  by  its  analysis  and  conclusions. 

D .  SOURCES 

1 .  The  Questionnaire 
a.  General 

At  the  center  of  the  research  work  done  for  the 
thesis  is  a  questionnaire  which  was  distributed  to  a  sample 
of  high-level  officials  in  the  U.S.  DOD  and  in  the  Israeli 
Air  Force  (IAF) .  The  purpose  of  the  questionnaire  was  to  ob¬ 
tain  inputs  associated  with  the  thesis  theme  from  people  who 
are  actually  involved  in  major  system  acquisition  decisions. 
Addressing  two  different  decision-making  milieus — the  American 
and  the  Israeli — provided  a  base  for  comparison,  in  which 


1As  one  can  notice,  no  distinction  is  made  here  between 
decision  authorities  (such  as  SecDef  or  (S)Sec)  and  recommending 
bodies  (such  as  DSARC,  (S)SARC).  The  reason  is  based  upon  the 
assumption  that  in  consolidating  their  recommendation,  the 
latter  should  reach,  in  a  sense,  some  sort  of  decision.  In 
this  respect  the  suggested  technique  may  apply  to  them  as 
well  as  to  the  formal  decision-makers. 
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the  American  setting  is  emphasized,  and  the  Israeli  serves 
as  a  reference. 

b.  The  Respondents  Sample 

The  sample  solicited  was  selected  with  reference 
to  positions  and  estimated  involvement  in  the  acquisition 
decision  process.  The  American  sample  consisted  of  38  offi¬ 
cials,  of  which  28  are  military  and  10  civilians.  Those  were 
spread  among  the  DOD  components  as  follows:  OSD — 13,  Navy 
Department — 16,  Air-Force  Department — 9.  Their  positions 
distributed  from  medium-level  staff,  up  to  Under  Secretary 
of  Defense  (R&E) . 

The  Israeli  sample  consisted  of  11  IAF  officers, 
including  two  deputies  to  the  IAF  Chief  of  Staff,  the  IAF 
Chief  of  Maintenance  and  Logistics,  IAF  HQs  department  heads 
and  staff  members . 

c.  The  Questionnaire  As  A  Reference 

The  original  questionnaire  and  its  full  analysis 
and  numerical  summaries  are  located  in  Appendix  A,  which  is 
a  complete  and  independent  unit.  Besides  the  direct  analysis 
of  the  questions,  Appendix  A  presents  a  variety  of  comments 
written  by  the  respondents,  some  of  which  are  unique.  The 
analysis  and  numerical  data  of  Appendix  A  serves  as  a  suppor¬ 
tive  reference  to  the  basic  analysis  of  the  thesis.  For  deeper 
insight  intc  the  respondents  perceptions,  reading  of  Appendix 
A  is  recommended. 

2 .  Interviews 

In  order  to  obtain  additional  and  better  insight  into 
the  acquisition  decision-making  issues  than  that  permitted 
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by  the  questionnaire,  several  interviews  took  place.  The 
interviewees  were: 

(1)  Dr.  Walter  B.  LaBerge,  Ex-Deputy  Under  Secretary  of 
Defense  (R&E) . 

(2)  Robert  F.  Trimble,  Deputy  Under  Secretary  of  Defense 
(R&E)  Acquisition  Policy  (acting) . 

(3)  Dr.  Robert  J.  Hermannn,  Special  Assistant  for  Intelli¬ 
gence  to  the  USD ( R&E) ,  and  until  a  week  before  the  inter¬ 
view — Assistant  Secretary  of  the  Air  Force  for  Research, 
Development  and  Logistics. 

(4)  BRIG.  GEN  Joseph  H.  Connolly,  Director,  Contracting 
and  Manufacturing  Policy,  DCS/Research,  Development  and 
Acquisition,  HQ  USAF. 

(5)  Frederick  G.  Fellowes,  Director,  Tactical  Air  Surface 
and  EW  Development  Directorate. 

In  order  to  discuss  Multi-Attribute  Utility  Theory  (MAUT) 1 
implementation  in  the  acquisition  decision-making  process, 
an  additional  interview  took  place.  The  interviewee  was: 

Prof.  Ward  Edwards,  Director  of  the  Social  Science  Research 
Institute  at  USC,  Los  Angeles,  Ca.  Prof.  Edwards  is  one  of 
the  leading  theorists  and  practitioners  of  MAUT.  His  publica¬ 
tions  served  as  major  references  to  this  thesis. 


In  the  literature,  various  names  and  acronyms  applied  to 
similar  ideas  are  found.  The  acronym  MAUT  may  be  an  abbre¬ 
viation  for  Multi-Attribute  Utility  Theory,  or  Technology, 
or  Technique .  In  addition,  there  can  be  found:  Multi- 
Attribute  Utility  Measurement  (MAUM) ,  Multi-Attribute  Utility 
Analysis  (MADA) ,  or  Simple  Multi-Attribute  Rating  Technique 
(SMART) . 
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3.  Literature 


The  thesis  also  refers  to  various  academic  sources 
as  well  as  to  official  publications. 

E.  ORGANIZATION 

This  thesis  consists  of  six  chapters.  Chapter  II  explores 
the  characteristics  of  the  present  acquisition  decision-making 
process,  and  reveals  its  problems  and  deficiencies  as  reflected 
from  the  literature  base,  the  ofncial  documentation  and  the 
perceptions  of  the  people  involved.  Special  emphasis  is  put 
on  examining  the  attitudes  toward  usage  of  quantitative  deci¬ 
sion  techniques,  and  the  extent  to  which  these  techniques  are 
actually  used.  Chapter  II  concludes  with  a  list  of  criteria 
for  a  decision  technique  intended  to  improve  the  current 
process . 

Chapter  III  examines  the  applicability  of  quantitative 
decision  techniques  for  top-level  acquisition  decision-making. 
Emphasis  is  put  on  studying  the  capability  of  those  techniques 
of  handling  subjective  judgment.  Among  the  decision  tech¬ 
niques  MAUT  is  found  as  most  suitable  for  acquisition  decisions. 

Chapter  IV  provides  the  theoretical  basis  of  MAUT,  which 
is  required  for  further  discussion  about  the  implementation 
aspects.  The  chapter  also  presents  clarification  of  the 
various  terms  used  interchangeably  in  MAUT-related  literature 
and  in  this  thesis  as  well. 

Chapter  V  highlights  the  major  issues  associated  in  MAUT 
implementation  as  a  tool  for  top-level  acquisition  decision¬ 
making.  The  chapter  provides  also  a  broader  view  on  MAUT 
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procedure  as  a  comprehensive,  ongoing  acquisition  evaluation 
framework.  The  chapter  demonstrates  to  what  extent  MAUT 
meets  the  criteria  set  in  Chapter  II. 

Chapter  VI  gives  the  overall  conclusions  and  general 
recommendations  of  the  thesis. 

Appendix  A  consists  of  a  detailed  analysis  of  the 
questionnaire . 
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II.  EXPLORATION  OF  THE  PRESENT  SITUATION 


A.  THE  GENERAL  NATURE  OF  THE  ACQUISITION  DECISION-MAKING 
PROCESS  AND  ASSOCIATED  PROBLEMS 

1 .  Formal  and  Informal  Process 

The  formal  acquisition  process  is  well  defined 
in  various  directives  and  instructions,  but  in  fact  is 
only  a  model  from  which  individual  programs  deviate. 

The  process  guidelines  represent  an  idealized  scheme 
for  selection  development  and  procurement  of  products... 

Along  with  the  formal  and  obvious  decision-makers 
there  are  li$s  conspicuous  ones  who  are  often  more 
influential.  [Fellowes,  1981] 

In  this  respect,  the  formal  acquisition  decision-making  process 
and  documentation  are  those  which  are  defined  by  DOD  Directive 
No.  5000.1  (1980),  DOD  Instruction  No.  5000.2  and  the  supple¬ 
mentary  regulations  within  the  Department  of  Defense  (DOD) 
components.  As  an  informal  process  are  considered  all  meetings, 
discussions,  influence  exertions,  writings,  etc.,  which  are 
not  directly  defined  and  required  by  the  above  mentioned  docu¬ 
ments  . 

Fellowes'  perception  is  supported  by  several  of  the 
questionnaire's  respondents.  Moreover,  some  of  them  even 
doubt  the  value  of  the  formal  decision  process  and  documenta¬ 
tion  in  the  "real  world"  decisions.  Says  one  respondent: 

Acquisition  process  documentation  is  of  limited 
value,  if  any.  [Appendix  A,  p.  165] 

This  study  addresses  both  the  formal  and  informal 
decision-making  processes.  The  suggested  decision-making 
technique  should  serve  not  only  as  a  decision  tool  for  the 
formal  decision  events,  but  also  resolve  some  deficiencies 
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caused  by  the  informal  process,  although  by  no  means  cure  them 
all.  The  informal  process  is  considered  as  an  inherent, 
inevitable  part  of  the  acquisition  decision-making  process, 
therefore  no  attempt  is  made  to  eliminate  it,  rather  to  better 
control  and  take  it  into  account  according  to  the  "rules  of 
the  game. " 

Most  of  the  references  about  the  present  situation 
are  taken  from  the  formal  process,  which  is  easier  to  identi¬ 
fy.  But  that  does  not  mean  the  formal  process  is  the  only 
one  to  be  addressed.  Nevertheless,  in  many  cases  the  formal 
process  situation  is  a  genuine  representative  of  more  essential 
approaches  to  decision-making,  which  are  not  necessarily  attached 
to  specific  current  procedures .  These  approaches  are  ad¬ 
dressed  in  this  thesis.  This  is  the  reason  why  the  recommen¬ 
dations  for  implementation  of  the  suggested  technique  (see 
Chapter  V)  are  stated  in  general  terms,  and  suggestions  for 
specific  changes  in  the  current  directives  and  instructions 
are  avoided.  It  is  true  also,  as  mentioned  by  some  of  the 
questionnaire's  commentators  [Appendix  A,  p.  164],  that  the 
"DSARC  Process"  is  subject  to  continuing  revisions.  But  since 
referring  mainly  to  the  underlying  approaches,  the  conclusion 
may  be  valid  in  a  new  acquisition  decision  process  as  well. 

2 .  Types  of  Major  Systems  Acquisition  Decisions 

The  formal  decision  points  in  the  acquisition  life 
cycle  are  identified  as  the  four  Milestones.  These  SecDef’s 
major  decisions  punctuate  a  continuous  process  of  evaluations, 
consolidation  of  sub-decisions  and  recommendations.  At  each 
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Milestone  an  authorization  to  proceed  to  the  next  phase  is 
approved  or  disapprove.  At  Milestones  I  and  II,  selection 
of  system  alternatives  is  done  as  well  [DOD  Dir.  5000.1,  1980, 
p.  4] .  Although  any  decision  is  “selection  of  particular 
alternative (s)  from  set  of  feasible  alternative  courses  for 
resolving  a  particular  problem"  [Krajewski,  1981,  p.  5],  the 
alternatives  of  the  Milestones  decisions  are  of  variable 
nature.  In  Milestone  I  a  selection  should  be  made  between 
alternative  design  concepts  for  development,  and  in  Milestone 
II  between  contending  specified  systems.  Since  in  most  cases 
system  selection  and  contractor  selection  are  equivalent,  it 
is  Milestone  II  at  which  Source  Selection  is  directed.  Mile¬ 
stone  III  decisions  usually  do  not  deal  with  alternative 
concepts  or  systems,  but  rather  with  a  single  system.  Here 
the  alternatives  are  mostly  variations  in  the  acquisition 
strategy,  i.e.,  trade-offs  between  performance,  cost  and 
schedule.  All  other  factors  are  in  most  cases  already  deter¬ 
mined.  Milestones  I  and  II  type  of  decisions,^"  are  those  in 
which  selection  between  different  system  concepts  (e.g.,  MX 
missile  vs.  B-l  bomber)  or  between  different  systems  of  the 
same  concept  (e.g.,  F-16  vs.  YF-17)  is  done,  and  a  broad  spec¬ 
trum  of  factors  may  still  affect  the  decision.  These  are  the 
decision  types  which  the  suggested  technique  (which  is  later 

^The  word  "type"  is  emphasized  since  the  statement  relates 
to  any  decision  at  any  level  which  meets  the  characteristics 
of  Milestones  I  and  II  decisions.  (Recall  the  author's  approach 
to  his  analysis  is  not  to  relate  it  to  any  specific  current 
decision  process,  which  is  subject  to  change.) 


introduced)  focus  on.  But,  of  course,  it  is  by  no  means 
restrained  only  to  this  type  of  decision.  Being  a  flexible 
and  adjustable  device  it  can  handle  other  decision  types  as 
well. 

3.  Acquisition  Decisions  As  Multi-Objective  Decisions 
The  previous  section  related  mainly  to  the  multiple 
alternative  nature  of  the  acquisition  decision.  This  section 
highlights  the  multi-objective  nature  of  those  decisions.  A 
broad  scope  of  objectives  must  be  achieved  by  top-level 
decision-makers.  Some  of  the  major  objectives  are: 

(1)  Maximize'*'  military  effectiveness. 

(2)  Minimize  copt  i 

(3)  Maximize  .  >al  utility. 

(4)  Maximize  •.•'tribal  and  external  political  benefits. 

(5)  Maximize  social  and  economic  benefits. 

These  objectives  are  in  fact  subdivided  through  several  levels 
in  an  hierarchical  order,  as  presented  in  more  detail  in  Chap¬ 
ter  V.  The  complexity  of  the  acquisition  decisions  is  not 
measured  solely  by  counting  its  objectives.  An  examination 
of  the  formal  process  reveals  how  many  items,  factors,  and 
criteria  may  be  taken  into  account.  For  example: 

(1)  The  Decision  Coordinating  Paper  (DCP)  which  provides 
the  primary  documentation  for  use  by  the  DSARC  in  arriving 
at  the  Milestone  recommendation  [DOD  Inst.  5000.2,  1980,  p.  6] 

*"The  term  "objective"  is  referred  here  as  the  "direction" 
in  which  one  should  strive  to  do  better  [Keeney  and  Raiffa, 
1976,  p.  34].  Consequently  the  words  "maximize"  and  "mini¬ 
mize"  indicate  direction,  and  not  specific  levels  of  minimum 
or  maximum  as  goals  to  achieve . 


presents  in  its  annexes  a  list  of  goals  and  threshold  of  20 
different  items  under  categories  such  as  cost,  schedule,  per¬ 
formance,  supportability  and  manpower.  In  addition  it  includes 
a  detailed  table  of  different  resources  and  life  cycle  cost 
(LCC)  components. 

(2)  The  Integrated  Program  Summary  (IPS)  which  summarizes 
the  implementation  plan  of  the  DOD  component  for  the  complete 
acquisition  cycle  (DOD  Inst.  5000.2,  end.  4,  p.  1]  requires 

5  mandatory  annexes  and  23  more  issues  to  be  addressed. 

(3)  A  chart  of  major  system  acquisition  life  cycle 

[C£C  Associate  Consultants,  19  80]  counts  16  decision  criteria 
for  Milestones  I  and  II. 

(4)  The  USAF  requires  about  35  data  items  for  its  Milestone 
Reviews,  consisting  of  a  variety  of  areas  (e.g.,  operational 
performance,  technology,  cost,  etc.,  (Department  of  the  AF, 
1979,  pp.  37-39]).  Even  if  we  assume  that  the  principle  of 
"management  by  exceptions"  is  valid  here,  as  is  the  case  in 
most  top-level  decision-making,  still  the  examination  of  the 
various  data  items  in  light  of  the  various  criteria  and  objec¬ 
tives  in  order  to  reach  the  ultimate  decision,  is  a  very 
difficult  task.  The  above  examples  relate  to  the  formal  inputs 
only.  The  task  is  much  more  difficult  when  informal  inputs, 
influences,  and  pressures  are  taken  into  account. 

The  difficulties  in  top-level  acquisition  decision¬ 
making  is  observed  not  only  by  analyzing  the  formal  documenta¬ 
tion  for  the  decision;  it  is  also  spelled  out  by  the  comments 
of  the  people  involved.  According  to  Connolly  (1981) , 
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sometimes,  when  a  program  office  comes  in  and  gives 
a  briefing,  it  is  very  difficult  to  really  grasp  the 
significant  or  distinguishing  factors.  You  need 
your  staff  to  be  involved,  to  look  into  that  in 
advance,  to  try  to  surface  the  discriminators. 

But  is  not  only  a  problem  of  grasping.  It  is  in  the  first 
place  a  problem  of  including  the  relevant  inputs  for  the  deci¬ 
sion,  and  according  to  one  of  the  questionnaire  commentators, 
"including  all  relevant  decision  factors  is  the  tough  part" 
[Appendix  A,  p.  200) .  Muddling  through  the  large  number  of 
pertinent  factors  leads  to  inefficiency  in  the  DSARC  process, 
while  on  one  hand  the  reviews  are  not  limited  to  key  issues 
and  many  insignificant  ones  are  raised.  On  the  other  hand, 
some  essential  issues  do  not  get  enough  attention  [Rice,  1978, 
pp.  33-34].  From  the  questionnaire  we  learn  that  although 
the  respondents  are  moderately  satisfied  with  the  manner  in 
which  alternatives  are  represented  and  compared  in  the  formal 
documentation,  they  indicate  to  a  considerable  extent  the 
difficulty  in  making  the  final  decision,  despite  the  refining 
process  prior  to  the  top-level  decision-making.  According 
to  Dr.  LaBerge  (1981)  , 

...no  serious  comparison  between  alternatives  can  take  ^ 
place  for  most  programs  under  the  existing  methodology. 

As  a  symptom  of  weakness  in  the  decision-making,  LaBerge 

points  out  that. 

In  fact,  the  actual  decision  made  by  the  senior  de¬ 
fense  officials  is  to  approve  or  disapprove  the 


^To  soften  this  statement  LaBerge  comments  that  "there 
are  exceptions  for  some  major  high  visibility  programs  such 
as  MX  missile  where  top  management  can  obtain  an  independent 
assessment  from  high  level  review  groups  such  as  the "Defense 
Science  Board"  [LaBerge,  1981]. 
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alternative  recommended  by  the  Service,  rather  than 
select  between  alternatives. 

This  tendency,  which  is  supported  by  the  questionnaire  findings 

[Appendix  A,  pp.  172-173]  might  be  a  wise  decision  tactic 

unless,  as  in  many  cases,  it  is  caused  by  the  fact  that 

. . .most  people  in  the  decision  process  neither  have 
studied  the  problem  in  detail,  nor  had  an  easily 
understood  impartial  breakout  of  the  issues.  [LaBerge, 

1981] 

4 .  The  Credibility  of  the  Decision  Alternatives 
Presentation 

Problems  of  the  present  acquisition  decision-making 
are  not  derived  only  from  their  complexity  as  being  of  a 
multi-objective  nature.  A  severe  criticism  about  the  deci¬ 
sion  presentation  objectivity,  the  alternatives  credibility 
and  the  extent  of  the  decision-maker  real  decision  option 
repeatedly  appears  in  the  comments  of  the  people  involved. 

Some  citations  that  illustrate  the  above  mentioned  notion 
include: 

...the  selection  is  never  less  than  clear-cut .. .The 
Service  assures  this... The  real  alternatives 
generally  do  not  appear — only  strawmen....  [Appendix 
A,  p.  170] 

The  arguments  and  evidence  are  usually  structured 
to  support  the  Service  selection  and  to  present 
the  decision  in  the  most  "clearcut"  form  permitted 
by  the  facts... By  the  time  such  reviews  are  held 
at  the  highest  levels  for  all  practical  purposes 
the  selection  has  been  made  and  any  other  choice 
or  selection  is  very  difficult  to  affect... 

[Appendix  A,  p.  170] 

Alternatives  other  than  the  preferred  alternative 
are  evaluated  to  gain  perspective  on  the  preferred 
one.  There  is  virtually  no  probability  any  other 
will  be  selected  [Appendix  A,  p.  172] 

In  many  cases  alternatives  besides  the  recommended 
one  are  presented  to  satisfy  formality .. .The 


process  is  too  far  committed  to  permit  reversal 
of  the  Service  position.  [LaBerge,  1981] 

These  quotes  clearly  speak  for  themselves  and  do  not  require 
further  interpretation.  Even  if  we  consider  that  they  do  not 
apply  to  all  cases,  and  do  not  represent  all  views,  still  they 
are  serious  enough  to  be  taken  into  account  in  searching  im¬ 
provement  of  the  present  situation. 

B.  THE  DIAGNOSTIC  APPROACH  TO  DECISION-MAKING:  A  DETAILED 
PROCESS  ANALYSIS 

1 .  The  Diagnostic  Approach — An  Overview 

The  previous  section  examined  the  present  acquisition 
decision-making  situation  from  general  aspects.  The  rest  of 
the  examination  dealing  with  more  detailed  characteristics, 
will  be  done  through  the  diagnostic  approach  to  decisions. 
According  to  this  approach,  decisions  are  examined  in  terms 
of  their  process  and  structure  (Keen,  1978,  p.  61].  The 
examination  of  the  decision  process  will  be  classified  accord¬ 
ing  to  five  main  schools  of  thought  which  are  found  in  the 
literature  on  decision-making.  These  are: 

(1)  The  rationalistic  view. 

(2)  The  satisficing  view. 

(3)  The  organizational  procedure  view. 

(4)  The  political  view. 

(5)  The  individual  differences  perspective.  [Keen,  1978, 
p.  63] 

Each  of  these  schools  suggests  that  its  view  should  be  ^he 
dominant  aspect  in  examining  a  decision  process,  and  any 
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suggested  decision-making  methodology  should  mainly  apply  to 
this  aspect,  and  concentrate  on  resolving  problems  derived 
from  it.  But  in  fact,  in  complex  decisions  all  those  aspects 
play  a  significant  role,  and  any  decision  methodology  aimed 
to  improve  such  complex  decision  processes  should  take  into 
account  requirements  derived  from  all  those  views.  That  is 
the  strategy  to  be  used  in  our  case  as  well.  Thus,  the  coming 
sections  will  analyze  the  present  acquisition  decision  process 
in  light  of  the  above  mentioned  five  schools  of  thought. 

2 .  The  Rationalistic  View 

a.  The  Theoretical  Definition 

This  is  a  normative  view  in  which  the  decision¬ 
maker  is  assumed  to  select  the  most  efficient  alternative, 
that  is,  to  maximize  the  amount  of  output  for  a  given  input. 

The  decision  criteria  are  predefined  and  the  objective  can  be 
formulated  in  a  quantitative  manner.  The  rationalistic  methods 
maximize  the  expected  utility  in  situations  where  there  is 
uncertainty  of  events  and  outcomes.  In  both  instances  there 
is  an  "objective"  method  of  arriving  at  a  solution:  given 
a  set  of  consequences  and  one's  utility  function,  the  choice 
is  automatic.  This  approach  assumes  a  rational,  completely 
informed  single  decision-maker  (Keen,  1978,  pp.  62,  64-65]. 

As  evidenced  by  Keen,  proponents  of  cost-benefit  analysis 
adopt  this  approach.  The  mam  shortcoming  of  this  approach 
is  that  once  one  defined  the  "rules  of  the  game",  he  is  bound 
to  their  mechanical-computational  results.  Or  as  Simon  (1957, 
pp.  241-260)  put  it,  the  logic  and  results  of  the  optimal 
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choice  remain,  even  where  they  are  descriptively  unrealistic. 

But  even  the  opponents  of  this  approach  do  admit  that  it  is 
of  value  in  highlighting  central  variables  in  the  decision¬ 
making,  and  enabling  good  communication  among  the  people 
involved  [Keen,  1978,  p.  64]. 

b.  Quantitative  Techniques  in  Use  for  Assisting 

Acquisition  Decision-Making 

At  the  foundation  of  the  rationalistic  approach 
rests  all  sorts  of  quantitative  models.  The  questionnaire 
reveals  relatively  low  propensity  to  use  quantitative  models, 
or  techniques  to  support  top-level  acquisition  decisions.  If 
they  are  used,  they  are  always  severely  limited,  using  very 
few  factors  as  the  only  basis  for  comparison  [Appendix  A, 
pp.  174-177],  Quantitative  techniques  are  hardly  used  as  an 
overall  evaluation  means  in  the  support  of  high  level  decisions. 
But  they  are  broadly  used  in  lower  levels  to  calculate  various 
complex  but  measurable  unit-dimension  factors,  such  as  cost, 
performance,  reliability,  maintainability,  manpower  and  the 
like.  In  his  response  to  the  questionnaire,  a  program  manager 
writes: 

At  the  program  level  we  tend  to  do  little  modeling 
outside  the  engineering  domain,  where  we  get  good 
correlation  factors.  Objective  modeling  forces 
discipline  but  is  not  our  tool  in  decision-making. 

[Appendix  A,  p.  181] 

Among  others,  the  specific  technique  that  lies 
at  the  center  of  this  thesis,  the  Multi-Attribute  Utility 
Theory  (MAUT) ,  is  used  on  some  occasions  in  the  acquisition 
evaluation  and  decision  process.  Moreover,  it  handles  not 
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only  measurable  dimensions,  but  also  subjective  ones — however 
on  a  realtively  narrow  scope.  It  is  used,  for  example,  as 
one  of  the  cornerstones  of  the  Mission  Operability  Assessment 
Technique,  (MOAT) .  This  technique  is  an  evaluation  methodology 
that  measures  the  operability  of  a  system,  or  subsystem,  in 
terms  of  operator  tasks  performed  during  mission.  It  is 
implemented  in  the  test  and  evaluation  of  the  F/A-18  [Pacific 
Missile  Test  Center,  1979] .  But  the  results  of  this  technique 
are  obviously  not  an  overall  evaluation. 

A  broader  scope  of  MAUT  implementation  is  observed 
in  the  Source  Selection  procedure.  Here  the  proposals  are 
evaluated  through  a  hierarchy  of  factors  and  subfactors,  whose 
top-level  categories  consist  of: 

(1)  General  quality  and  responsiveness  of  the  proposal. 

(2)  Organization,  personnel  and  facilities. 

(3)  Technical  approach. 

(4)  Cost.  [Procurement  Associates,  1972,  pp.  XVIII-13 — 30] 
But  this  is  also  not  comprehensive  enough  to  base  a  major 
system  selection  on,  as  mentioned  by  the  above  source  itself 
(p.  XVIII-30)  : 

For  large  programs,  the  final  choice  may  be  made  at 
the  top  of  the  agency  level,  or  in  the  case  of  major 
weapons  systems  at  the  Department  of  Defense  level, 
after  evaluation  by  many  subordinate  groups.  The 
technical  and  cost  evaluation  is  only  part  of  the 
consideration.  The  industrial  mobilization  require¬ 
ments  and  the  effect  of  the  award  of  the  contract 
on  the  cost  of  related  Government  contracts  are 
only  a  few  of  the  many  factors  that  must  be  considered. 

[The  underlines  made  by  the  thesis  author,  R.G.) 

Another  version  of  evaluation  procedure  for  Source 
Selection  is  introduced  in  a  newer  source — the  Navy  Program 
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Manager  Guide  [Naval  Material  Command,  1980,  pp.  2-27 — 2-33). 
Here  the  top-level  categories  are: 

(1)  Life  Cycle  Cost, 

(2)  Contractor  evaluation, 

(3)  Effectiveness. 

(See  Figure  II-l.)  But  again,  the  scope  is  still  too  narrow 
for  an  overall  evaluation  since  it  excludes  factors  such  as 
political,  economic,  social,  environmental,  etc.,  which 
undoubtedly  have  significant  impact  on  major  systems  selection 
(though  the  extent  depends  on  the  case) .  The  evaluation  ap¬ 
proach  chosen  by  the  Navy's  Source  Selection  procedure  fully 
correlates  with  the  basic  premise  of  this  thesis.  Its  only 
"disadvantage"  is  that  it  is  contract-oriented  and  not  neces¬ 
sarily  ultimate  "in-house"  (within  the  DOD  top-levels)  selec¬ 
tion-oriented.  Hermann  (1981)  emphasizes  this  notion  by 
indicating  that  Source  Selection  evaluation  procedure  is 
needed  primarily  to  conduct  the  relationships  between  the 
Government  and  potential  contractors . 

It  is  a  requirement  of  the  contractual  world  to 
have  a  deterministic  kind  of  means  for  choosing 
between  competitors.  But  high  levels,  who  look 
for  the  right  thing  to  do,  do  not  have  this 
constraint. 

Hermann  means  that  top-level  does  not  have  to  use  the  same 
procedure  in  their  decision-making  (he  advocates  not  to  use 
quantitative  models  at  all) .  But  the  ultimate  decision, 
using  different  evaluation  procedures  and  more  important, 
based  on  significant  considerations  which  are  external  to  the 
Source  Selection  evaluation,  should  not  necessarily  correlate 
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Figure  II-l.  Structuring  and  Weighting  Evaluation  Cri¬ 
teria  for  Source  Selection  (taken  from  Navy 
Program  Manager  Guide  [Naval  Material 
Command,  1980]) 
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the  Source  Selection  outcomes.  That  might  be  the  case 
especially  with  major  systems  selection,  in  which  political 
and  similar  factors  play  an  important  role.  Since  Source 
Selection  procedure  can  be  a  basis  for  a  legal  suit,  the 
possibility  of  conflicting  ultimate  decisions  puts  under  ques¬ 
tion  the  very  usage  of  this  formal  procedure  for  major  systems 
selection.  Thus,  the  existing  Source  Selection  evaluation  is 
not  only  short  of  being  a  top-level  decision-making  tool,  but 
it  also  contains  an  inherent  "catch,"  which  questions  its 
usefulness  even  as  a  Source  Selection  tool  for  major  systems 
(at  least  for  those  which  are  politically  or  socio-economically 
sensitive) . 

A  view  from  another  angle  on  the  use  of  quantita¬ 
tive  decision  techniques  is  noted  by  one  of  the  questionnaire's 
respondents,  referring  to  the  (S)SARC’s  review  sessions: 

The  AFSARC  does  not  use  models  for  decision-making 
directly.  We  are,  however,  briefed  on  the  use  of 
decision  aids  and  quantitative  methods  used  by  the 
staff  in  developing  recommendations.  [Appendix  C, 
p.  179] 

It  seems  that  this  is  the  case  in  most  top-level  decision-making 
occasions,  in  which  no  overall  evaluation  technique  centers 
the  decision-making,  not  to  speak  of  any  kind  of  interaction 
between  the  decision-maker  and  this  assisting  technique.  In 
spite  of  all  that  has  been  said  above,  there  are  some  "candles 
in  the  dark."  One  of  them  is  found  in  the  Navy  Program  Manager's 
Guide  [Naval  Material  Command,  1980,  pp.  3-22  —  3-25] .  This 
guide  does  not  only  present  MAUT  model  for  Source  Selection, 
but  it  also  suggests  using  a  similar  model  for  risk  management. 
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This  tool  is  supposed  to  be  used  by  the  program  manager  and 
his  staff  in  evaluating  program  alternatives  which  involve 
limited  information  and  uncertainties.  The  suggested  model 
includes  a  considerable  element  of  subjective  judgment.  But 
no  word  is  said  about  using  that  model  for  the  major  deci¬ 
sions,  and  no  involvement  of  high-level  decision-makers  in 
the  model  construction  or  its  parameters  elicitation  is  men¬ 
tioned.  It  is  perceived  as  sort  of  an  "in-house"  tool.  If 
one  judges  from  the  responses  of  the  two  program  managers 
from  the  questionnaire  sample,  the  actual  use  of  such  a 
mechanism  in  the  program  office  is  very  limited  [Appendix  A, 
p.  181] .  Another  example  is  presented  in  the  technical  report 
"Decision  Analysis  of  Advanced  Scout  Helicopter  Candidates" 
[Decisions  and  Designs,  1980],  ordered  by  the  U.S.  Army  Avia¬ 
tion  Research  and  Development  Command  to  support  its  position 
at  ASARC.  This  report  contains  an  analysis  of  thirteen  Ad¬ 
vanced  Scout  Helicopters  (ASH)  candidates  and  some  mixture 
of  those  candidates.  The  analysis  evaluates  the  candidates 
on  the  basis  of  a  wide  scope  of  factors  of  which  the  top 
levels  are:  military  worth;  life  cycle  costs;  attainability, 
force  structure  and  personnel  impacts,  and  RSI  impacts  [Deci¬ 
sions  and  Designs,  1980,  p.  11] .  This  technical  report  uses 
MAUT  as  its  evaluation  methodology.  It  is  based  heavily  on 
subjective  judgments  and  refers  sparingly  to  political  and 
economic  considerations,  as  expected  from  a  model  being  de¬ 
veloped  for  the  Service  level.  But  the  basic  approach  is 
exactly  the  one  this  thesis  advocates  to  be  used  also  in 
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decisions  made  above  the  Service  level.  There  would  be  no 
requirement  for  this  thesis  if  such  models  were  customarily 
used  for  top-level  decision-making.  But  research  done  for 
this  thesis  indicates  that  they  are  not.  Furthermore,  there 
is  considerable  reluctance  towards  using  such  models.  At  the 
least,  there  is  concern  over  improper  implementation  and  biased 
models'  construction.  The  remainder  of  the  Advanced  Scout 
Helicopter  case  illustrates  the  above  statement.  In  a  phone- 
talk  to  Michael  Donnell  (1981) ,  one  of  the  two  authors  of  the 
referred-to  report,  the  following  facts  were  acknowledged: 

(1)  The  analysis  team  worked  exclusively  with  the  program 
manager,  a  Colonel,  and  his  staff,  and  did  not  interact  with 
other  higher  levels  of  participants  or  the  actual  decision¬ 
making  group  members . 

(2)  The  Colonel  "knew  exactly  what  he  and  his  boss  wanted," 
and,  in  some  cases,  determined  the  construct  of  the  model  that 
was  in  conflict  with  the  analysis  team's  opinion.  That  caused 
some  insignificant  value-dimension  to  be  located  at  the  top 

of  the  hierarchy,  while  others,  perhaps  much  more  important, 
receive  inappropriately  low  location.  [The  last  comment  is 
not  a  quote,  rather  the  thesis  authors'  interpretation.  R.G.] 

(3)  Eventually  the  decision  that  has  been  made  was  other 
than  that  recommended  by  the  model  "because  of  budget  con¬ 
straints  that  arose,  and  the  'politics'  of  the  decision-makers 
who  had  strong  prior  views  about  the  selection,  which  could 
hardly  be  affected  by  the  model's  analysis."  In  an  answer  to 
a  question,  Donnell  said  that  in  theory  both  elements — cost 
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changes  and  participants  views — could  be  included  in  the  model 
but  (a)  The  analysis  has  been  kind  of  a  "one  shot"  type,  and 
therefore,  has  not  been  up-dated  according  to,  say,  cost  con¬ 
straints  that  were  not  originally  taken  into  account;  (b)  The 
analysis  team  had  not,  in  fact,  access  to  the  decision-group 
members,  and  thus  their  views  revealed  only  at  late  stages, 
when  the  analysis  and  its  use  got  out  of  the  teams  control; 

(c)  The  analysis  was  supposed  to  be  mainly  cost-ef f ectivess 
related,  and,  thus,  intentionally  did  not  put  great  emphasis 
on  all  kinds  of  politics-related  aspects. 

The  deficiencies  observed  in  this  case  can  be 
summarized  as  follows: 

(1)  Improper  representation  of  participants  involved  in 
the  decision. 

(2)  Biased  model  structure. 

(3)  Although  broader  in  scope  than  most  models  used,  still 
limited  for  presenting  comprehensive  evaluation  for  the  final 
selection. 

(4)  Discontinuity  and  lack  of  evaluation  model  updating. 

These  deficiencies  are  repeated  in  many  examples  through  the 
thesis.  The  conclusions  of  this  case  will  be  addressed  among 
the  rest  in  implementation,  analysis,  and  recommendations  (see 
Chapter  V) . 

The  questionnaire  verifies  the  observation  that 
models  or  other  sorts  of  quantitative  techniques  are  used 
only  to  a  small  extent  to  assist  top-level  acquisition  decision¬ 
making,  and  whenever  they  are  used,  they  are  of  limited  scope 
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[Appendix  A,  pp.  174-178] .  Furthermore,  a  considerable  number 
of  the  respondents  admit  that  they  have  never  used  a  quantita¬ 
tive  model  for  acquisition  decisions  of  any  kind  at  any  level 
[Appendix  A,  p.  184]. 

c.  Reservations  of  Quantitative  Techniques  as  a 

Top-Level  Acquisition  Decision-Making  Tool 

The  first  and  foremost  reservation  of  quantita¬ 
tive  techniques  as  an  overall  evaluation  and  comparison  means 
for  acquisition  decisions,  is  the  substantial  doubt  about 
their  ability  to  handle  subjective,  intuitive,  judgmental 
considerations.  Trimble  (1981)  notes  as  an  example  the  MX 
missile  case,  in  which  he  points  out  the  political,  environ¬ 
mental  and  sociological  impacts  associated  with  its  basing 
concept,  as  such,  "do  not  lend  themselves  to  any  type  of 
predictive  quantitative  model."  But  it  is  not  only  a  matter 
of  the  issues  under  question,  it  is  also  a  matter  of  the 
decision-makers  themselves  who,  as  described  by  Trimble,  are 
"very  capable  and  experienced  people,  but  usually  unstructured 
individuals,  who  do  not  follow  'a  rule  of  bock'.  How  would 
they  be  willing  to  associate  themselves  with  a  structured 
decision  process?"  he  asks. 

Even  more  extreme  in  his  aversion  toward  quanti¬ 
tative  decision  techniques  is  Hermann  (1981) ,  who  says  that 
a  lot  of  subjective  factors  are  not  amenable  to  numerical 
treatment.  Hermann  says: 

I  always  found  it  difficult  philosophically  to  under¬ 
stand,  how  a  decision-maker  could  submit  himself  to 
a  deterministic  formula,  in  a  decision  which  is 
fundamentally  imbeded  in  highly  subjective  factors. 


Part  of  this  approach  is  supplemented  in  the  literature. 

For  example,  Quade  (1979,  p.  10)  in  his  book,  Policy  Analysis 

for  Public  Decisions,  which  basically  advocates  quantitative 

decision  techniques,  warns: 

...there  are  always  considerations  that  cannot  be 
handled  quantitatively,  maybe  not  analytically,  or 
even  systematically,  and  there  m<=.-'  be  problems 
with  no  solution.  In  the  end,  pox  tics  and  intui¬ 
tive  judgment  must  rule. 

Question  10  of  the  questionnaire  addresses  this 
very  issue,  stating  that  "there  is  no  use  for  a  decision-making 
model  to  support  top-level  decision-makers,  because  they  base 
their  judgment  mainly  on  subjective  experience,  perception  and 
intuition,  which  cannot  be  rationally  "managed."  The  opinions 
about  this  statement  are  split:  43%  agree  with  the  statement 
and  51%  disagree,  and  5%  have  no  definite  idea  [Appendix  A,  pp. 181-184] . 

Edwards  (1981)  agrees  with  the  observation  that 
acquisition  decisions  are  fundamentally  subjective,  but  he 
also  thinks  that  there  is  a  fundamental  misconception  with 
respect  to  quantitative  techniques,  at  least  as  far  as  MAUT 
is  concerned.  His  explanation  of  the  deep  reservations  that 
the  above  quoted  people  have  towards  those  techniques  is  that 
the  standard  kind  of  models  they  have  been  exposed  to  were 
either  performance  models,  or  cost-benefit  models.  While 
performance  models  deal  with  objective  issues  almost  explicitly, 
the  traditional  versions  of  the  cost-benefit  models  reduce 
everything  to  dollar  units,  which  certainly  takes  the  subjec¬ 
tivity  out.  According  to  Edwards,  the  respondents  are  just 
unfamiliar  with  techniques  which  are  especially  designed  to 
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handle  subjective  judgment  and  intuition  (though  by  no  means 
to  replace  or  generate  them) .  To  conclude,  it  should  be 
stated  that  inspite  of  the  reservations,  all  the  interviewed 
officials  and  the  questionnaire's  respondents  do  see  some 
degree  of  merit  in  a  specified  use  of  quantitative  model  as 
a  decision  aid.  This  premise  will  be  introduced  in  more  de¬ 
tail  in  the  following  chapters. 

Another  objection  to  quantitative  techniques  is 
attributed  to  the  previously  mentioned  biased  alternatives' 
presentation.  As  it  is  put  bitingly  by  one  of  the  question¬ 
naire's  commentators, 

...use  of  such  a  model  is  phony  as  a  'three  dollar  bill  I 
It's  a  travesty  on  honest  quantif ication. . .The  use  of 
a  model  by  the  Service  makes  everyone  feel  better  and 
maintain  the  area  of  'ob jectivity ’ .  It  has  nothing 
to  do  with  the  Service  choice.  [Appendix  A,  p.  184] 

Others  warn  against  the  possibility  of  a  model  structured 

to  support  a  previously  selected  alternative  [Appendix  A, 

p.  184J. 

Another  worry  about  usage  of  decision  models  is 

expressed  by  the  following  quote: 

...if  a  decision-maker  embraced  a  model,  but  then 
disagreed  with  the  model  outcome  he  may  be  baring 
his  belly  to  his  foes.  [Appendix  A,  p.  206] 

But  Edwards  (1981)  observation  that,  in  most  cases,  decision¬ 
makers  are  not  so  insecure,  is  supported  by  the  majority  of 
the  questionnaire  respondents,  who  do  not  view  this  possibility 
severely  enough  to  cause  them  to  refrain  from  using  decision 
models  [Appendix  A,  pp.  206_21lJ • 

Finally,  it  should  be  emphasized  that  the  purpose 
of  this  section  has  been  to  disclose  problems,  weaknesses  and 
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difficulties  in  quantitative  decision  techniques.  Therefore 
it  addresses  the  criticism  and  apprehension  towards  them. 

But  there  were  also  positive  reactions  which  are  introduced 
later  in  this  thesis. 

3 .  The  Satisficing  View 

a.  The  Theoretical  Definition 

This  approach  focuses  on  how  a  decision-maker  can 
most  effectively  use  limited  knowledge  and  skills  [Simon,  1957. 
pp.  241-260] .  It  highlights  the  constraints  imposed  by  "bounded 
rationality,"  the  emphasis  on  heuristic  "rules  of  thumb,"  and 
searching  for  solutions  that  are  "good  enough."  This  approach 
is  based,  in  most  cases,  on  a  descriptive  model  of  the  decision 
process.  The  goal  is  to  improve  the  existing  solution,  "not 
to  vainly  seek  for  an  optimum." 

b.  Application  in  the  Acquisition  Decision-Making 
It  seems  that  in  many  of  the  present  acquisition 

decision-making  the  satisficing  view  is  the  prevailing  ap¬ 
proach.  Connolly's  (1981)  description  of  his  own  pattern  of 
thought  during  a  decision  event  applies  to  a  great  extent  to 
this  approach.  Hermann's  (1981)  clear  preference  of  a  descrip¬ 
tive  decision  model,  "a  clear  logic  in  our  natural  language," 
corresponds  to  the  satisficing  approach  as  well.  But  some 
questions  are  raised:  is  the  satisficing  selection  sufficient 
for  critical  decisions?  Do  decision-makers  really  reach  the 
bounds  of  their  "limited  knowledge  and  skills"?  The  answer 
of  this  thesis  to  both  of  these  questions  is  "no".  There 


can  be  ways  to  achieve  more  than  just  "good  enough"  decisions. 
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There  can  be  a  technique  with  which  decision-makers  would  be 
able  to  better  exploit  their  own  knowledge  and  skills  to 
reach  a  decision  based  on  the  data  presented  to  them. 

4 .  The  Organizational  Procedure  View 

a.  The  Theoretical  Definition 

This  concept  of  decision-making  seeks  to  under¬ 
stand  decisions  as  the  output  of  standard  operating  procedures 
invoked  by  organizational  subunits.  The  emphasis  is  on  the 
formal  and  informal  structure  of  the  organization,  its  standard 
operating  procedures  and  channels  of  communication  [Keen,  1978, 
pp.  63,69-70] . 

b.  Application  in  the  Acquisition  Decision-Making 

It  is  clear  that  for  major  systems  the  final  choice 
made  by  the  SecDef  is  based  on  evaluations  made  by  many  sub¬ 
ordinate  groups  [Procurement  Associates,  1972,  p.  XVIII-30] . 
LaBerge  (1981)  says  that, 

...for  most  programs  top-levels  inevitably  must  rely 
upon  evaluations  done  by  their  subordinates,  and 
this  is  passed  down  to  the  middle  levels  of  the 
Services,  where  the  motivations  are  influenced  much 
more  by  capability  than  by  cost. 

The  reliance  on  lower  levels  mentioned  by  LaBerge, 
is  challenged  by  the  questionnaire's  respondents  in  Q 7  [Appen¬ 
dix  A,  p.  170]  where  54%  disagree  with  the  statement  that  "the 
less  the  selection  is  clearcut,  the  greater  is  the  decision¬ 
maker's  tendency  to  rely  on  lower  level  analysis  and  recom¬ 
mendations."  But  yet,  a  significant  portion  of  35%  think  to 
the  contrary.  In  any  case,  it  is  quite  obvious  that,  as 
noted  by  one  of  the  questionnaire  respondents, 


...staff  members  can  be  very  influential  in  the^r 
principal's  ultimate  decision  by  carefully  structur¬ 
ing  and  selectively  presenting  the  alternatives. 

[  H—  2  ] 

This  brings  us  back  to  the  notion  of  biased  presentations 
and  false  alternatives  ( "strawmen" )  mentioned  at  the  beginning 
of  this  chapter.  This  emphasizes  a  "fact  of  life,"  that 
through  the  inevitable  organizational-hierarchical  pattern 
of  the  decision  process,  interest  groups  may  try  to  influence 
the  decision  to  their  side.  That  can  be  done  in  the  worst 
case  by  "a  hatchet  job  done  on  the  alternatives"  [Appendix  A, 
p.  170] ,  or  by  a  genuine  perception  which  does  not  necessarily 
correlate  with  that  of  the  decision-maker.  As  an  example  of 
the  latter  can  be  taken  LaBerge's  observation  that  the  ser¬ 
vices  attribute  much  more  importance  to  capability  (performance) 
than  to  cost.  This  observation,  although  strongly  rejected 
by  the  Services  (for  example,  by  Fellowes  (1981)),  is  supple¬ 
mented  by  a  Report  To  The  Congress  by  the  Comptroller  General 
(1979),  stating  that, 

...there  are  many  programs  where  extraordinary  perform¬ 
ance  requirements  have  been  the  driving  force  behind 
high  costs. 

The  controversy  is  further  disclosed  by  a  somewhat  cynical 

comment  made  by  one  of  the  questionnaire  respondents: 

. . .quantitative  rationale  for  the  decision  could 
be  most  useful  in  countering  accountants  budgeteers 
and  other  non-operationally  experienced,  but  none¬ 
theless  influential  decision-makers.  [ H— 2 ] 

This  presentation  of  the  problematics  in  the  real  life  deci¬ 
sion  process  correlates  to  the  "classical"  deficiencies  at¬ 
tributed  by  the  literature  to  decision  procedures,  as  observed 
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through  the  organizational  view.  These  deficiencies  are: 

(1)  Parochial  perceptions  and  group  pressures. 

(2)  Inflexibility  and  adherence  to  routine  procedures. 

(3)  Losing  the  overall  picture  by  breaking  decisions  into 
small  pieces. 

(4)  Sequential  attention  to  goals,  according  to  the 
organization's  routine.  [Keen,  1978,  pp.  63,69-70] 

These  deficiencies  should  be  addressed  in  any  suggested  improve¬ 
ment  to  current  decision  practice. 

5 .  The  Political  View 

a.  The  Theoretical  Definition 

According  to  the  political  view,  decision-making 
is  viewed  as  a  bargaining  process  between  the  formal  and  in¬ 
formal  organizational  units  and  personalities.  A  premium  is 
placed  on  understanding  the  realities  of  power  and  on  the 
compromises  and  strategies  necessary  to  mesh  the  interest  and 
constraints  of  the  "actors"  in  the  decision  process  [Keen, 

1978,  p.  63] .  Political  aspects  of  a  decision  can  be  leveled 
in  three  tiers: 

(1)  The  personal  tier,  in  which  personalities  affect  the 
decision.  For  example,  a  strong,  charismatic  person,  equipped 
with  good  bargaining  skills,  confidence,  and  quick  reaction 
may  influence  a  decision  towards  his  desired  direction. 

(2)  The  organizational  tier,  in  which  bargaining  and  power- 
play  take  place  between  organizations  or  groups  of  partici¬ 
pants  (e.g.,  the  Services,  OSD,  Industry,  Congress  and  others). 
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(3)  The  national  tier,  in  which  national  political  inter¬ 
ests  are  involved. 

b.  Applications  in  the  Acquisition  Decision-Making 

No  doubt,  all  three  tiers  of  the  political  view 

apply  to  the  acquisition  decision-making  process.  The  role 

of  the  personalities  involved  is  clearly  described  by  one  of 

the  questionnaire's  respondents: 

The  personnel  experience,  and  preferences  of  senior 
officials  in  key  positions,  plays  heavily  in  the 
decision  process .. .The  skills  of  the  program 
sponsor  to  communicate  the  value  of  his  program 
has  great  impact  on  the  decision.  [Appendix  A,  p.  183] 

Another  respondent,  referring  the  usage  of  models  in  review 

sessions,  states: 

The  review  sessions  are  never  so  emotionless  as  to 
deal  with  a  dispassionate  review  of  alternatives... 

The  model  will  be  totally  perverted.  It  is  not  an 
issue  of  models — it's  a  matter  of  power  and  emotion 1 
[Appendix  A, p. 177,183] 

Fellowes  ( 1981)  describes  the  frustration  feeling 
experienced  sometimes  at  the  Service  level,  while  viewing  a 
"pure, "  well-defined  weapon  system  diverted  from  its  original 
requirements  as  a  result  of  political  pressures.  But  he  also 
agrees  that  there  is  no  way  to  keep  politics  out,  since  it  is 
an  inherent  component  of  the  decision  process.  Trimble  (1981) 
counts  the  politics  as  one  of  the  three  prominent  considera¬ 
tions  in  top-level  acquisition  decisions,  together  with  the 
need  and  the  cost. 

Hermann  (1981)  realizes  the  weight  of  the  personal 
politics  relative  to  the  direct  elements  of  the  decision  by 
defining  the  AFSec  acquisition  decision  sessions  as  a  "very 
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complex  social  event,  but  a  very  simple  information  event. 

The  information  conveys  are  very  crude  and  primitive."  The 
questionnaire  resondents  estimate  out  of  57  systems  acquisi¬ 
tion  cases  that  on  the  average,  the  political  and  socio¬ 
economic  issues  weight  about  50%  of  SecDef  acquisition  decision 
considerations.  This  weight,  though  gradually  reducing  to 
match  the  decision  level,  remains  substantial  even  in  lower 
levels  [Appendix  A,  pp.  231-  236]. 

The  previous  paragraphs  reflect  the  impact  of 
political  considerations  on  the  acquisition  decision.  These 
considerations  are  an  integral  part  of  a  democratic  process, 
and  thus  cannot  and  should  not  be  taken  out  of  it.  The 
decision-maker  should  strive  for  an  objective  decision  in  the 
sense  that  it  represents  unbiased  public  values.  A  "public 
value"  is  assigned  to  an  outcome  by  a  public,  usually  by  means 
of  some  public  institution  that  does  the  evaluating.  This 
amounts  to  treating  "a  public"  as  a  sort  of  organism  whose 
values  can  be  elicited  by  some  appropriate  methods  already  in 
use  to  elicit  individual  values.  From  this  point  of  view, 
the  decision-making  is  associated  with  finding  the  appropriate 
adaptation  of  those  methods — an  adaptation  that  will  take 
into  account  the  individual  (or  grouped)  disagreements,  indi¬ 
vidual  (or  grouped)  differences  in  relevant  expertise,  and 
the  existing  socio-political  structures  [Edwards,  1977,  p.  248]. 
In  other  words,  the  political  power  play  should  affect  the 
decision  in  a  balanced,  objective  and  ordered  manner  as  much 
as  possible.  The  various  interests  should  be  properly 
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represented,  and  differences  variation  in  resulting  decisions 
stemming  from  personalities  should  be  reduced. 

6 .  The  Individual  Differences  Perspective 

a.  The  Theoretical  Definition 

This  perspective  concentrates  on  the  individual 
decision-maker  and  his,  or  her,  probelm  solving  and  informa¬ 
tion  processing  behavior  [Keen,  1978,  p.  63].  There  have  been 
two  broad  approaches  in  the  individual  differences  perspective: 

(1)  The  cognitive  complexity,  which  argues  that  there  is  an 
optimal  balance  of  information  input  for  any  one  individual. 

(2)  The  cognitive  style,  which  emphasizes  the  style  of  the 
information  and  the  decision  process.  For  example,  "syste¬ 
matic  thinkers"  tend  to  approach  a  problem  by  structuring  it 
in  terms  of  some  method  which  if  followed  through  leads  to  a 
likely  solution  (that  corresponds  to  the  previously  mentioned 
rationalistic  view) .  On  the  other  hand,  "intuitive  thinkers" 
usually  avoid  committing  themselves  in  this  way;  their  strategy 
is  more  one  of  hypothesis-testing  and  trial  and  error  [McKenney, 
1974,  p.  81] . 

b.  Applications  in  the  Acquisition  Decision-Making 
The  cognitive  complexity  approach  suggests  that 

the  amount  of  information  presented  is  unique  to  any  individual 
decision-maker.  But  dealing  with  top-level  decision-makers, 
one  basic  need  is  almost  common  for  all,  and  that  is — simplicity . 
Edwards,  for  example,  through  years  of  developing  his  MAUT 
concepts  puts  a  great  emphasis  on  the  simplicity  of  his  tech¬ 
nique  (see  for  example  Edwards  (1971,  pp.  119-129;  1977,  p.  250!  and 
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Edwards  and  Newman  (1980,  Executive  Summary,  p.  1)).  While 
it  lacks  the  theoretical  elegance  of  techniques  proposed  by, 
for  example,  Raiffa  (1968,1969)  or  Keeney  (1972),  it  has  the 
great  advantage  of  being  easily  learned  and  used  by  a  busy 
decision-maker  or  member  of  decision-making  staff  or  organi¬ 
zation.  Keen  (1978,  p.  97)  describes  a  desired  managerial 
decision  model  as  being  "very  simple  and  crude,  rather  than 
mathematically  sophisticated.  It  is  often  based  on  heuristic 
rules  and  standard  procedures  for  analysts." 

Connolly  (1981)  requires  a  decision  model  to  be 
uncomplicated  and  easy  to  understand.  Hermann  (1981)  reveals 
a  special  aversion  towards  computerized  decision  models: 

"My  own  experience,"  he  says,  "is  that  a  lot  of  computer 
models  tend  to  destroy  the  intuition  you  have  left."  He  is 
supported  by  one  of  the  questionnaire ' s  commentators ,  who 
says  that  "the  structured  decision  process  is  a  useful  tool... 
but  it  must  emphatically  not  be  plugged  into  a  computer..." 
[Appendix  A,  p.  227].  Thus,  any  decision  presentation  to 
top-level  decision-maker (s)  should  be  easy  to  grasp,  even  by 
people  whose  expertise  is  not  necessarily  in  the  very  details 
of  the  system  alternatives  or  in  sophisticated  modeling  proce¬ 
dures.  Models  that  have  their  logic  "hidden"  within  a  computer 
or  can  be  understood  only  by  the  experts  who  built  them,  have 
a  little  value  for  top-level  decision-makers,  if  any. 

The  cognitive  style  approach  is  reflected  in  several 
written  comments  to  the  questionnaire  as  well  as  in  the  inter¬ 
views.  According  to  those  comments,  any  decision  technique  to 
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be  used  must  not  only  be  acceptable  to  the  final  decision¬ 
makers,  but  also  be  brought  in  at  early  stage  in  time  to  be 
modified  by  their  comments,  and  to  assure  data  base  etc.,  is 
organized  to  their  "taste"  [Appendix  A,  p.  227,  and  Connolly, 
1981]  . 

C.  CHAPTER  CONCLUSION 

1 .  General 

As  a  conclusion  of  this  chapter  a  list  of  require¬ 
ments  for  a  desired  decision  technique  is  presented.  This 
may  be  considered  as  a  guideline  in  searching  for  such  a 
technique,  or  as  a  check-list  in  examining  a  proposed  one. 

The  list  accumulates  requirements  from  all  vievs  of  the  deci¬ 
sion-making  process  in  order  to  exploit  their  individual  advan¬ 
tages,  and  avoid  or  reduce  their  weaknesses.  In  that  sense, 
these  are  requirements  for  some  hypothetical,  ideal,  and 
probably  utopian  decision  technique  because  no  technique 
can  satisfy  them  all.  Furthermore,  not  everyone  would  accept 
all  these  requirements  as  desirable.  However  such  a 
list  can  serve  as  a  good  basis  to  start  the  search  for 
improvement  in  the  current  decision  process. 

2 .  A  List  of  Requirements  from  a  Decision  Technique 

a.  Requirements  Elicited  from  the  General  Nature 
of  the  Acquisition  Decision-Making 

The  decision  technique  should: 

(1)  Take  into  account  the  informal  decision  process  and 
actors,  as  well  as  the  formal  ones. 

(2)  Be  capable  of  handling  multi-objective  and  multi¬ 
factor  decisions. 


45 


\3)  Focus  on  the  significant  issues,  scan  the  less  signi¬ 
ficant,  and  eliminate  the  irrelevant  ones. 

(4)  Improve  the  decision-makers'  capability  to  evaluate 
and  select  between  the  decision  alternatives. 

(5)  Reduce  biases  in  decision  presentations,  and  increase 
objectivity  of  alternative  presentation. 

(6)  Maintain  continuity  and  evaluation  model's  updating. 

b.  Requirements  Elicited  from  the  Rationalistic  View 

The  decision  technique  should: 

(1)  Be  a  comprehensive  means  for  overall  evaluation. 

(2)  Be  quantitative  by  nature  but  maintain  the  capability 

of  handling  subjective,  intuitive  and  judgmental  considerations. 

(3)  Be  implemented  in  such  a  way  that  will  reduce  the 
probability  of  the  model  being  used  to  counter  the  decision¬ 
maker  . 

c.  Requirements  Elicited  from  the  Satisficing  View 

The  decision  technique  should: 

(1)  Use  descriptive  elements  and  "rules  of  thumg,"  espe¬ 
cially  for  the  judgmental  components  of  the  technique. 

(2)  Exploit  to  the  maximum  the  knowledge  and  skills  of  the 
decision-makers . 

d.  Requirements  Elicited  from  the  Organizational 
Procedure  View 

The  decision  technique  should: 

(1)  Reduce  the  dependence  of  top-level  decision-maker  on 
lower  level  evaluations. 


(2)  Spell  out  the  importance-weights  attributed  to  vari¬ 
ous  factors  at  lower  levels  (and  by  that,  perhaps,  disclose 
some  of  the  intentional  and  unintentional  biases  in  the 
decision  presentation) . 

(3)  Exploit  the  organizational  structure  for  partial  and 
intermediate  evaluations  and  decisions,  but  make  sure  that 
they  are  consolidated  to  an  overall  picture  at  the  final 
decision  phase. 

e.  Requirements  Elicited  from  the  Political  View 
The  decision  technique  should: 

(1)  Give  an  "equal  right"  of  representation  to  the  various 
people  and  organizations  who  have  stakes  in  the  decision. 

(2)  Articulate  political  interests,  including  "selfish" 
ones. 

(3)  Compensate  for  personal  differences  in  charisma,  bar¬ 
gaining  skills  or  presentation  talents. 

f.  Requirements  Derived  from  the  Individual  Differ¬ 
ences  Perspective 

The  decision  technique  should: 

(1)  Be  simple  and  intelligible. 

(2)  Have  sound  and  visible  underlying  rationale. 

(3)  Be  acceptable  to  the  decision-maker,  and  directed  by 
him  from  the  early  stages. 

These  requirements  will  provide  guidance  in  the 
search  for  an  actual  decision  technique  and  its  implementation 
in  top-level  acquisition  decision-making. 


47 


III.  SEARCH  FOR  DECISION  METHODOLOGY 


A.  THE  BASIC  APPROACH  TO  THE  SOLUTION 
1 .  The  Essential  Dilemma 

The  essential  underlying  dilemma  in  the  search  for 
top-level  acquisition  decision-making  methodology  is  whether 
or  not  the  key  approach  should  be  quantitative.  The  reser¬ 
vations  towards  quantitative  techniques  as  an  overall  top- 
level  evaluation  device  have  been  introduced  in  detail  in 
Chapter  II. 

At  the  core  of  those  reservations  rests  the  lack  of 
confidence  in  the  capability  of  quantitative  techniques  to 
incorporate  subjective  judgment.  The  ultimate  conclusion  is 
not  to  use  quantitative  techniques  at  all  as  a  comprehensive 

i 

decision  basis.  Hermann  (1981)  supports  that  idea: 

I  am  not  favorably  impressed  with  complex  modeling 
as  a  basis  for  important  weapons  systems  decisions. 

A  clear  logic  in  our  natural  language  is  the  most 
important  and  preeminent  need. 

But  there  are  opposing  opinions  as  well,  such  as  LaBerge's 

(1981) ,  who  says: 

( 

The  principal  difficulty  is  that  the  selection  of 
alternatives  has  little  quantitative  basis,  which 
could  be  disputed. . .A  quantitative  decision  tool 
is  absolutely  desirable,  and  might  bring  about  an 
essential  change  in  the  quality  of  the  acquisi¬ 
tion  decision-making. 

Thus,  it  is  this  issue  of  whether  or  not  quantitative  tech¬ 
niques  are  compatible  for  top-level  acquisition  decision¬ 
making,  which  this  sectio  i  concentrates  on.  Special  attention 
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is  paid  to  the  subjective  judgment  handling  capabilities  of 
those  techniques. 

2 .  The  Role  of  Judgment  in  Acquisition  Decisions 

Keen  (1978,  pp.  86-87)  divides  decisions  into  three 
categories : 

(1)  Structured. 

(2)  Semistructured. 

(3)  Unstructured. 

A  similar  term  (well-structured,  moderately-structured,  and 
ill-structured)  are  used  by  Dunn  (1981,  pp.  103-105)  in  speci¬ 
fying  public  policy  problems.  It  seems  that  acquisition  deci¬ 
sions  fall  somewhere  inbetween  the  second  and  the  third 
categories  of  problems  as  defined  by  the  two  above  mentioned 
authors.  They  fit  Keen's  definition  for  semistructured  deci¬ 
sions  very  well,  i.e.,  "decisions  where  managerial  judgment 
alone  will  not  be  adequate,  perhaps  becuase  of  size  of  the 
problem  or  its  complexity.  On  the  other  hand  the  model  or 
data  alone  are  also  inadequate  because  the  solution  involves 
some  judgment  and  subjective  analysis."  But  the  acquisition 
decisions  and  especially  their  political  aspects,  may  fit  as 
well  Dunn's  definition  for  ill-structured  public  policy  prob¬ 
lems,  in  which  the  main  characteristics  are  "conflict  among 
competing  goals,  and  decision-making  which  involves  high  levels 
of  conflict  among  competing  stakeholders"  [Dunn,  1981,  pp.  104- 
105].  The  acquisition  decisions  in  many  cases  also  fit  Dunn's 
definition  for  moderately-structured  problems,  since  they 


are  usually  involving  one  or  few  decision-makers,  and  a 


relatively  limited  number  of  alternatives.  Dunn’s  charac¬ 
teristic  of  moderately-structured  problems,  which  views  these 
problems  as  "reflecting  consensus  on  clearly  ranked  goals" 
[Dunn,  1981,  p.  104]  can  be  achieved  in  some  cases.  Although 
the  starting  point  is  usually  the  conflicting  attitudes  upon 
values,  after  some  iterations  of  interaction  with  stakeholders, 
an  agreement  or  even  consensus  may  be  achieved.  The  last 
statement  is  approved  by  Edwards  ( 1981)  who  reached  in  some 
practical  occasions  through  face-to-face  interaction,  stake¬ 
holders'  general  recognition  of  a  single  "value  tree"^  as 
being  a  fair  representation  of  their  perceptions.  In  the 
acquisition  decisions  some  of  the  factors  are  completely 
structured  and  objectively  measurable,  some  require  subjec¬ 
tive  judgment  imposed  on  objective  data,  and  some  consist  of 
pure  judgmental,  intuitive  analysis.  At  the  top-levels  the 
latter  have  a  considerable  relative  significance.  Some  people 
involved  in  the  actual  decision-making  process  emphasize  this 
relative  high  significance  of  judgment  by  saying,  "...Most 
decisions  are  fundamentally  subjective..."  [Hermann,  1981], 
or  "...In  my  experience  most,  if  not  all,  acquisition  deci¬ 
sions  are  a  matter  of  judgment"  [Appendix  A,  p.  203 ] .  The 
role  of  judgment  in  the  acquisition  decisions  fits  Keen's 
(  1978,  p.  97)  observation  about  the  role  of  judgment  in  semi- 
structured  decisions: 


'''This  term  will  be  defined  in  Chapter  IV,  p.  79  -  81  . 
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The  judgement  is  necessary  either  to  recognize  or  decide 
what  constitutes  the  problem,  or  to  create  alternatives, 
or  to  choose  a  solution.  The  judgment  may  define  the 
nature  of  the  variables  that  are  considered  or  the 
values  that  are  put  on  the  known  variables. 

3 .  Compatibility  of  Quantitative  Techniques  For 
Top-Level  Acquisition  Decision-Making 

As  a  consequence  of  the  previous  section's  analysis, 
the  first  thing  to  do  is  to  look  at  the  capability  of  quanti¬ 
tative  techniques  to  handle  judgment.  Peck  and  Scherer  (1962, 
pp.  543-580)  approached  the  issue  of  weapon  system  performance 
evaluation  empirically.  They  asked  whether  "it  is  possible 
to  combine  these  diverse  considerations  into  a  single  concep¬ 
tual  scheme  which  permits  us  to  evaluate  the  overall  effec¬ 
tiveness  of  a  weapon  program."  Their  analytical  conclusion 
was  that  measurement  in  the  conventional  sense  is  impossible, 
but  it  may  be  possible  to  determine  which  program  is  better 
executed  than  others  and  thus  to  establish  a  ranking  of  pro¬ 
grams  from  best  to  worst.  They  recognized  that  ranking  poses 
a  basic  theoretical  problem  since  weapon  programs  performance 
"is  presumably  multidimensional."  Their  answer  was  that 
"somehow  the  numerous  dimensions  underlying  programs  perfor¬ 
mance  must  be  weighted  and  combined  if  a  unidimensional  rank¬ 
ing  is  to  be  obtained."  Through  their  experiment  Peck  and 
Scherer  proved  that,  based  on  judgment  only,  the  individuals 
of  the  control  group  were  able  to  express  their  preferences 
and  importance  weights  consistently  and  to  express  them  in 
numerical  terms.  Keeney  and  Raiffa  (1976,  p.  40)  cite  another 
example  of  a  similar  experiment  done  by  Huber,  Sahney  anu  rord 
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(1969,  pp.  483-489) ,  who  concluded  that  the  results  strongly 
indicated  professionals  could  develop  and  reliably  use  sub¬ 
jective  evaluation  models.  This  is  not  to  say  that  every 
human  intangible  consideration  involved  in  a  decision  can  be 
quantified.  Rather,  the  major  consequences  of  these  consid¬ 
erations  may  be  quantified  in  terms  of  ranking  order  and  rela¬ 
tive  impact  on  the  decision.  As  mentioned  by  Keeney  and 
Raiffa  (1976,  p.  40),  "the  literature  in  psychometrics  is 
replete  with  examples  that  establish  such  scales  of  subjec¬ 
tive  index."  The  above  mentioned  experiments  support  Edwards' 
(1981)  confidence  in  techniques  especially  designed  for  handl¬ 
ing  subjective  judgment  such  as  MALJT.  His  confidence  is  based 
on  a  considerable  experience  of  himself  and  others.  According 
to  Edwards,  letting  the  decision-maker  create  his  own  param¬ 
eter  estimates  is  appealing  on  an  intuitive  basis.  It  gives 
the  decision-makers  a  feeling  that  they  have  more  control  over 
the  process.  Additionally,  Dawes  and  Corrigan  (1974)  show 
that  these  estimates  compare  favorably  to  those  derived 
empirically  in  regression  models. 

By  ruling  out  the  obstacle  of  the  "prejudice"  about 
quantitative  techniques  incapability  of  handling  subjective 
judgment,  the  way  is  open  to  exploit  the  generally  agreed 
advantages  of  those  techniques,  i.e.,  being  a  means  for 
rigor,  discipline  and  communication  in  complex  decisions. 
Furthermore,  Edwards  and  Newman  (1980,  Chapter  1,  p.  3)  empha¬ 
size  that  the  willingness  to  accept  subjectivity  into  evalua¬ 
tion,  combined  with  the  insistence  that  judgments  be  numerical, 
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serves  several  useful  purposes:  First  it  partly  closes  the 
gap  between  intuitive  and  judgmental  evaluations  and  the 
more  quantitative  kind;  it  makes  coexistence  of  judgment  and 
objectivity  measurement  within  the  same  evaluation  easy  and 
natural.  Second,  it  opens  the  door  to  easy  combination  of 
complex  concatenations  of  values.  For  example,  it  bridges 
over  the  distinction  often  made  by  researches  between  process 
evaluation  and  outcome  evaluation.  Third,  it  can  greatly 
shorten  the  time  required  for  an  evaluation  to  be  carried  out. 

But  a  question  is  still  valid:  assuming  that  quanti¬ 
tative  techniques  are  capable  of  handling  subjective  judgment-- 
are  they  really  required  and  desired  for  top-level  acquisition 
decisions?  Recall  LaBerge's  response  to  this  question  which 
has  already  been  introduced  in  the  opening  section  of  this 
chapter.  His  strongly  favoring  attitude  is  sound.  Other 
interviewees  like  Trimble  (1981) ,  Hermann  (1981)  and  Connolly 
(1981),  in  spite  of  their  reservations,  think  that  quantita¬ 
tive  models  may  have  merit,  and  can  be  helpful  even  to  deci¬ 
sion-makers  such  as  the  Defense  Acquisition  Executive  (DAE) 
or  SecDef.  The  questionnaire  respondents,  although  far  from 
being  enthusiastic,  lean  towards  recognition  of  quantitative 
models  as  a  useful  decision  tool,  including  for  top-level 
decision-making  [Appendix  A,  p.  194] .  The  compatibility  of 
such  models  to  acquisition  decisions  is  much  more  sound  from 
the  responses  that  refer  to  a  specific  example  of  decision 
technique < namely  MAUT  [Appendix  A,  pp.  186-188]  .  Many  of  the 
reservations  relate  to  deficiencies  external  to  the  models 
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themselves,  such  as  the  decision  environment  ("The  review 
sessions  are  never  so  emotionless  to  deal  with  a  dispassion¬ 
ate  review  of  alternatives"  [Appendix  A,  p.  177 ) .  Others 
refer  to  the  actual  way  models  are  currently  implemented  ("The 
scope  of  models  limited,"  [Appendix  A,  p.  177]),  as  the  main 
*l£.:v  in  models'  usefulness,  rather  than  their  inherent  fea¬ 
tures  [Appendix  A,  pp.  179-194]. 

An  indirect  conclusion  about  the  contribution  to 
decision-making  of  quantitative  techniques  can  be  drawn  from 
the  review  of  the  Israeli  respondents,  who  consistently  indi¬ 
cated  via  the  questionnaire  a  greater  inclination  towards  usage 
of  such  techniques,  compared  to  their  American  colleagues. 

At  the  same  time  they  demonstrate  a  greater  confidence  in 
their  ability  to  "close"  decisions.  The  linkage  seems  to  be 
clear . 

Several  citations  from  the  literature  may  contribute 
to  the  recognition  of  quantitative  techniques  as  being  help¬ 
ful  decision-making  tools.  Cohen  (1977),  an  Israeli  Air  Force 
high-level  officer  refers  to  the  issue  of  overall  evaluation 
of  alternatives  in  his  analysis  of  Headquarters  decision¬ 
making.  Cohen  suggests  two  methods  of  decision-making,  both 
based  on  breaking  evaluation  of  the  alternatives  into  factors. 
One  consists  of  weighted  scores  assigned  subjectively  to  fac¬ 
tors  and  then  aggregated  to  final  preference  (which  is 
basically  MAUT) .  Edelman  (1965)  provides  an  interesting 
comparison  of  the  quality  of  managerial  decisions  in  a  compe¬ 
titive  bidding  situation,  with  and  without  the  use  of  a 
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judgment  based  model.  In  seven  tests  managers-plus-model  won 
the  bid,  while  managers  alone  won  only  three.  The  model  was 
extremely  simple  and  did  little  more  than  to  assist  the 
managers'  own  estimating  procedures. 

England  (1975,  pp.  435-438),  dealing  with  selection 

of  an  adequate  source  of  supply  states: 

...the  decision  to  place  a  certain  volume  of  business 
with  a  supplier  is  always  based  on  some  rationale. 

The  art  of  good  purchasing  is  to  make  the  reasoning 
behind  that  decision  as  sound  as  possible. 

These  phrases  precede  an  introduction  of  various  types  of 
evaluations — some  objective  and  ethers  subjective,  which  are 
done  by  separate  factors.  Each  factor  is  assigned  an  impor¬ 
tance  rate  (weight) ,  and  each  alternative  is  scored  based  on 
each  factor.  The  overall  evaluation  is  derived  from  aggrega¬ 
tion  of  the  products  of  weight  times  score  for  each  alterna¬ 
tive  separately.  (Again,  this  is  basically  MAUT.) 

A  similar  method  is  suggested  by  Archibald  (1976,  p.  63) 
in  referring  to  high  technology  project  evaluation.  Archi¬ 
bald's  evaluation  is  done  according  to  the  following  guidelines: 

(1)  Priority  factors  are  listed  for  each  project  with  a 
value  of  1  to  10  assigned  to  each  factor  by  several  key  managers. 
These  scores  are  summarized  and  compared. 

(2)  Using  the  comparative  scores  and  other  information  not 
easily  quantified,  the  "Project  Priority  Review  Board"  meets 
sequentially  and  agrees  on  the  relative  priority  to  be  accorded 
each  project.  Archibald  does  not  mention  specifically  a  usage 
of  factor  weights,  but  does  mention  that  "the  relative 
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importance  of  the  factors  will  vary,  depending  on  the  organi¬ 
zation  and  the  type  of  project  involved." 

The  Brandaid  system,  designed  by  Little  (1975),  is 
an  interactive  Decision  Support  System  (DSS)  that  supports 
the  decision  process  in  commercial  marketing,  planning,  and 
estimating  overall  profitability.  The  main  feature  of  the 
Brandaid  system  is  its  emphasis  on  the  role  of  judgment  in 
the  decision  process.  Little  found  that  managers  generally 
have  a  good  understanding  of  the  dynamics  of  the  market,  or 
at  least  of  the  interrelationships  among  its  components, 
taken  in  pairs.  But  they  are  not  capable  of  determining  the 
full  interactions  of  different  components  simultaneously. 
Little  strongly  recommends  using  judgment  obtained  in  an  or¬ 
ganized  way  from  more  than  one  person.  According  to  Little, 
using  a  quantitative  technique  Brandaid  encourages  managers 
to  become  more  explicit  and  analytic  in  their  problem  solving, 
but  it  still  strongly  relies  upon  their  experience,  knowledge 
base  and  personalized  judgments. 

A  major  conclusion  of  LeGrow's  thesis  (1976,  p.  2) 
is  that  judgmental  scaling  techniques  are  presently  more 
valuable  for  measuring  capability  of  weapon  systems  than  more 
computerized  procedures.  According  to  LeGrow,  among  those 
judgmental  scaling  techniques  the  Multi-Attribute  utility 
scaling  affords  the  best  opportunity  for  ratio  comparisons 
of  weapons  capability,  as  being  able  to  handle  complex  phe¬ 
nomena  composed  of  both  intrinsic  performance  characteristics 
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and  external  factors,  like  operating  environment,  operator 
proficiency  or  the  technological  capacity  of  the  user  [LeGrow, 
1976,  p.  44] . 

A  similar  approach  is  chosen  Sherwin  and  Laurance 

(1979,  pp.  377-387)  in  their  search  for  better  methodology 

for  arms  transfers  and  military  capability  evaluation.  They 

find  that  the  techniques  generally  known  as  "subjective 

measurement  techniques"  are  the  most  suitable  for  the  task 

and  among  them  the  MAUT  is  preferred.  They  emphasize  the 

judgment  handling  capability  of  MAUT  by  saying: 

Basing  a  capability  measure  on  multiattribute  utility 
theory  capitalizes  on  the  notion  that,  to  date, 
human  insight  remains  the  most  reliable  means  for 
synthesizing  the  interrelations  among  a  complex  set 
of  international  relations  variables,  and  that  one 
means  of  indexing  an  otherwise  intangible  concept 
is  to  tap  the  collective  judgments  of  human  experts. 

According  to  this  notion,  humans,  by  developing  an 
intuitive  expertise,  determine  what  factors  are 
salient;  they  interpret  relevant  information;  and 
by  making  weighted  judgments,  they  differentially 
integrate  several  types  of  information  to  form 
subjective  evaluations  regarding  key  concepts. 

Quade  (1979,  p.  48)  recognizes  that  for  public  policy 
questions,  where  social  and  political  considerations  tend  to 
dominate,  models  to  be  used  to  predict  the  consequences  of 
choice  depend  more  and  more  directly  on  judgment  and  intuition. 
According  to  Quade,  an  explicit  model,  scientific  or  other¬ 
wise,  introduces  structure  and  terminology  to  a  problem,  and 
provides  a  means  for  breaking  a  complicated  decision  into 
smaller  tasks  that  can  be  handled  one  at  a  time.  It  also 
serves  as  an  effective  means  of  communication,  enabling  the 
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participants  in  the  study  to  make  their  judgments  in  a  con¬ 
crete  context  and  in  proper  relation  to  the  judgment  of 
others.  Moreover,  through  feedback — the  results  of  computa¬ 
tion  in  an  analytical  model  or  the  countermoves  in  a  game — 
the  model  can  help  the  analyst  and  the  experts  on  whom  they 
depend  to  revise  their  earlier  judgments  and  thus  arrive  at 
a  clearer  understanding  of  their  subject  matter  and  of  the 
problem. 

All  this  variety  of  excerpts  from  the  theoretical 
literature  and  the  “real  life"  experience  comes  to  support 
and  strengthen  the  actual  “actors’"  somewhat  hesitating  recog¬ 
nition  of  quantitative  decision  techniques  as  a  useful  tool 
for  top-level  acquisition  decision-making.  The  following 
are  the  summarized  conclusions  resulting  from  the  above 
citations : 

(1)  There  are  in  existence  quantitative  techniques  which 
are  capable  of  handling  subjective  judgment  (but  do  not  gen¬ 
erate  it;  this  is  an  exclusive  task  for  human  beings) . 

(2)  For  acquisition  decisions,  which  are  of  semi-structured 
nature,  judgment  based  quantitative  techniques  might  be  of 
great  help. 

(3)  The  secondary  characteristics  of  the  quantitative  models, 
i.e.,  the  separation  of  tasks  and  the  provision  of  a  systematic, 
efficient  and  explicit  way  to  focus  judgment  and  intuition — 
are  of  crucial  importance.  They  provide  a  route  for  tracing 
out  the  major  consequences  of  choice. 

In  addition  to  the  above  three  major  conclusions,  many 
of  the  cited  sources  give  some  clues  with  respect  to  a 
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preferred  decision  technique  by  pointint  out  indirectly 
or  explicitly  the  basic  concept  of  MAUT. 

Q.  MAUT— THE  PREFERRED  TECHNIQUE 

1 .  Limitations  of  the  Traditional  Weapon-Systems 

Comprehensive  Evaluation  Techniques 

The  two  most  prevailing  weapon-systems  evaluation 
techniques  were  cost-effectiveness  and  cost-benefit  analysis. 
Both  meet  the  requirement  of  being  quantitative  techniques 
which  are  widely  based  on  subjective  judgment.  But  they 
still  have  some  significant  limitations  that  put  their  fit¬ 
ness  under  question  as  an  overall  evaluation  device  for 
top-level  decision-makers. 

Cost-effectiveness  is  a  form  of  systems  analysis  in 

which  alternative  actions  or  systems  under  consideration  are 

compared  in  terms  of  two  of  the  consequences:  dolar,  or 

resource  cost,  and  the  effectiveness  associated  with  each 

alternative.  The  effectiveness  of  an  alternative  is  measured 

by  the  extent  to  which  that  alternative,  if  implemented,  will 

attain  the  desired  objective  iQuade,  1979,  p.  25].  Even  the 

definition  reveals  the  limited  scope  of  cost-effectiveness 

models.  Basically,  they  can  handle  one  objective,  which  is 

usually  for  weapon  systems — maximizing  the  combat  (operational) 

effectiveness.  As  emphasized  by  Quade, 

Whereas  we  are  often  able  to  use  cost-effectiveness 
to  rank  competing  alternatives  for  the  same  goal, 
we  cannot  use  it  to  compare  alternatives  that  seek 
different  goals — to  decide,  say,  the  best  over¬ 
all  use  of  our  money  when  we  have  several  long- 
range  objectives  in  mind. 
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But  the  latter  is  often  the  case  in  early  stages  of  the 
acquisition  decision  (e.g.,  concept  selection  at  Milestone 
I)  . 

Dunn  (1981,  p.  251)  also  emphasizes  the  limited 

scope  of  cost-effectiveness  approach  by  saying: 

...cost-effectiveness  analysis  epitomizes  technical 
rationality,  since  it  attempts  to  determine  the 
utility  of  policy  alternatives,  but  without  re¬ 
lating  their  consequences  to  global  economic 
efficiency  or  aggregate  social  welfare. 

But  again,  the  latter,  plus  political  considerations  (which 

are  not  mentioned  here  explicitly),  are  in  many  cases  very 

crucial  considerations  in  the  major  systems  acquisition 

decisions . 

Edwards  (1981)  views  the  limited  scope  of  cost-effec¬ 
tiveness  from  the  cost  aspect.  According  to  Edwards,  this 
technique  aggregates  all  cost  components  to  be  a  unit  dimen¬ 
sion  (a  single  figure  represents  all  costs) .  But  in  real 
life  people  attribute  different  importance  to  different  cost 
components  and,  thus,  are  willing  to  consider  them  separately. 
(For  example,  one  who  is  short  of  cash  cares  more  about  the 
down  payment  than  about  operating  costs  although  both  are 
components  of  the  total  life  cycle  cost  (LCC) ) .  Presumably, 
this  is  the  type  of  model  a  commentator  had  in  mind  when  he 
said: 

...the  model  [used  for  acquisition  decisions,  R.G.] 
is  always  severely  limited,  for  example,  it  might 
use  acquisition  cost  as  only  basis  for  comparison. 

[Appendix  A,  p.  177] 


Cost-benefit  is  a  broader- scoped  evaluation  technique 
compared  to  cost-effectiveness.  By  expressing  both  the  bene¬ 
fits  and  costs  in  monetary  units,  the  cost-benefit  technique 
allows  evaluation  and  comparison  of  programs  designed  to  ac¬ 
complish  widely  differing  tasks,  on  the  same  unit  base  [Quade, 
1979,  p.  26].  Quade  points  out  that  the  great  disadvantage  of 
cost-benefit  analysis  is  that  it  is  very  difficult  to  perform 
satisfactorily.  It  is  really  hard  to  express  every  human 
benefit  in  dollar  terms,  or  as  it  is  put  by  Prest  and  Turvey 
(1965)  : 

...one  can  view  cost-benefit  analysis  as  anything 
from  an  infallible  means  of  reaching  the  new  Utopia, 
to  a  waste  of  resources  in  attempting  to  measure 
the  unmeasurable . 

Dunn  (1981,  p.  345)  supports  this  criticism  by  emphasizing 
that  dollar  amounts  are  not  perceived  as  equally  significant 
by  different  persons,  and  therefore  dollars  are  an  inadequate 
measure  of  responsiveness.  (This  problem  of  limited  inter¬ 
personal  comparisons  often  means,  for  example,  that  income  is 
an  inappropriate  measure  of  individual  satisfaction  and  social 
welfare) .  Edwards  (1981)  thinks  the  reduction  of  everything 
to  dollar  units  in  the  traditional  versions  of  cost-benefit 
models  "certainly  takes  the  subjectivity  out." 

In  conclusion,  the  traditional  judgment-based,  quanti¬ 
tative  decision  techniques  have  sound  limitations  when  used 
for  a  comprehensive  evaluation  of  major- systems  acquisition 
alternatives.  Thus,  a  better  technique  should  be  looked  for, 
as  is  actually  done  in  the  next  section. 
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2 .  MAUT  as  a  Most  Suitable  Technique 


So  far  throughout  this  thesis  quite  a  few  examples 
of  various  versions  of  the  MAUT  concept  have  been  introduced. 
These  examples  cover  various  areas,  but  all  of  them  have  much 
in  common  with  system  acquisition  decisions,  although  none 
of  them  precisely  covers  the  whole  scope  of  factors  involved 
in  those  decisions.  Here  is  a  brief  reminder  of  the  examples 

(1)  The  formal  Source  Selection  procedure  (see  Chapter  II, 
p.  27)  . 

(2)  Peck  and  Scherer's  weapon  systems  relative  performance 
evaluation  (see  Chapter  III,  p.  51). 

(3)  Cohen's  decision-making  method  in  the  Headquarters 
(see  Chapter  III,  p.  54) . 

(4)  England's  selection  source  of  supply  (see  Chapter  III, 
p-  55)  . 

(5)  Archibald's  high  technology  projects  evaluation  (see 
Chapter  III,  p.  55)  . 

(6)  LeGrow's  weapon  systems  capability  evaluation  (see 
Chapter  III,  p.  56). 

(7)  Helm's  mission  operability  assessment  (see  Chapter 
II,  p.  27)  . 

(8)  Sherwin's  arms  transfers  and  military  capability 
evaluation  (see  Chapter  III,  p.  57). 

(9)  Donnell's  decision  analysis  of  advanced  scout  heli¬ 
copter  candidates  (see  Chapter  II,  pp.  31-33)  . 

(10)  Navy  Program  Manager's  Guide's  risk  management  model 


(see  Chapter  II,  p.  28-29) . 


In  the  literature  MAUT  is  distinguished  by  its  capa¬ 
bility  of  handling  a  variety  of  factors  such  as  technical, 
political,  economic,  social,  or  environmental,  in  various 
types  of  selections  between  similar  and  dissimilar,  old  and 
new,  certain  and  uncertain  alternatives.  In  order  to  stress 
MAUT's  versatility,  here  is  an  additional  list  of  MAUT  actual 
applications.  Some  are  practical  applications,  others  are 
experimental.  In  all  these  cases  MAUT  has  proven  to  be 
capable  in  circumscribing  a  complex  decision.  The  examples 
are  as  follows: 

(1)  Airport  development  for  Mexico  City  [Keeney  &  Raiffa, 

1976,  pp.  436-472] . 

(2)  Selecting  nuclear  power  plant  sites  in  the  Pacific 
Northwest  [Keeney  &  Nair,  1977]. 

(3)  Selecting  dams  sites  around  Phoenix,  Az .  [Edwards, 
1981]  . 

(4)  Selection  between  future  energy  supply  alternatives 
(nuclear,  coal  and  combined  geothermal  and  conservation  pack¬ 
age)  for  Southern  California  [Social  Science  Research  Insti¬ 
tute,  1981]  . 

(5)  Land  use  regulation  on  the  Californian  Coast  [Edwards, 

1977,  pp.  256-265] . 

(6)  Planning  research  program  for  the  Office  of  Child 
Development  (OCD)  of  the  U.S.  Department  of  Health,  Education 
and  Welfare  (HEW)  [Edwards,  1977,  pp.  265-267]. 

Finally,  MAUT  was  highly  preferred  by  the  question¬ 
naire  respondents  over  any  other  decision  technique  for 
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acquisition  decisions.  MAUT  was  selected  by  50%  of  the 
Americans  and  55%  of  the  Israeli's,  while  none  of  the  other 
contending  approaches  exceeded  15%.  Furthermore,  referring 
to  a  concrete  MAUT  example,  the  respondents  demonstrated 
high  confidence  in  the  compatibility  of  this  technique  to 
acquisition  decisions,  including  at  top-levels.  This  time 
their  attitude  was  much  more  favorable  than  their  reaction 
to  the  general  use  of  quantitative  models  for  top-level 
decision-making  [Appendix  A,  p.  186-188] .  In  addition,  the 
respondents  view  several  favorable  side-effects  of  the  MAUT 
procedure,  some  of  them  highly  correlated  to  those  already 
mentioned  in  citations  from  the  general  literature.  These 
side-effects  are: 

(1)  fr'AUT  allows  better  communication  and  understanding 
among  the  people  involved  in  the  decision-making. 

(2)  MAUT  has  the  potential  to  focus  discussion  on  the 
important  issues. 

(3)  MAUT  assists  decision-makers  to  consolidate  their 
thinking,  and  forces  rigor  into  discussions. 

But  the  respondents  also  warned  against  some  poten¬ 
tial  pitfalls  in  using  MAUT: 

(1)  Including  irrelevant  data  items. 

(2)  Overlooking  significant  issues. 

From  the  positive  side,  the  respondents  expressed 
their  confidence  that  MAUT  does  not  suppress  the  decision¬ 
maker's  intuition,  bias  his  judgment  or  restrict  his  decision 
freedom.  Finally,  many  of  the  respondents  emphasized  that 


64 


although  it  may  serve  as  a  comprehensive  evaluation  tool,  and 
give  a  useful  insight  into  the  choice  being  made,  MAUT  proce¬ 
dure  never  would  and  never  should  be  the  ultimate  determinant 
of  the  decision.  It  is,  after  all,  only  a  tool,  and  thus 
should  not  dominate  the  decision.  The  responsibility  always 
rests  on  the  shoulders  of  the  decision-maker,  and  no  model 
can  be  used  as  an  excuse  when  he  has  to  account  for  his 
decision  [Trimble,  1981,  and  Appendix  A,  pp.  202-212]. 

C.  CONCLUSIONS 

The  conclusions  of  this  chapter  are  the  following: 

(1)  The  acquisition  decisions  are  of  semi-structured  nature, 
and  combine  objectively  measured,  structured  data,  with 
highly  subjective  judgmental  considerations. 

(2)  For  such  kinds  of  decisions  judgmental  scaling  tech¬ 
niques  might  be  of  great  use.  The  judgment  handling  capabili¬ 
ties  of  these  techniques  are  well-established  through  analysis, 
experiments  and  actual  practice. 

(3)  Among  those  techniques  MAUT  is  the  preferred  one.  Its 
merits  are  recognized  by  a  variety  of  authors,  users,  and — 
perhaps  most  important--by  the  questionnaire  respondents  who 
represent  the  defense  systems  acquisition  decision-making 
community. 

(4)  MAUT  has  some  limitations  and  potential  pitfalls  that 
should  be  noticed  in  its  implementation. 

These  conclusions  allow  us  to  turn  to  a  basic  acquain¬ 
tance  with  MAUT,  which  takes  place  in  the  next  chapter. 
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IV.  MULTI-ATTRIBUTE  UTILITY  THEORY  (MAUT) 


A.  GENERAL 

The  purpose  of  this  chapter  is  to  provide  the  reader  a 
basic  acquaintance  with  MAUT,  in  order  to  understand  its 
basic  concepts.  Since  this  thesis  addresses  mostly  the  simple 
forms  of  MAUT  because  it  views  simplicity  as  an  essential 
requirement  for  top-level  decision  applications,  no  extensive 
introduction  of  MAUT  has  been  undertaken.  The  user  may,  if 
necessary,  turn  to  the  branched  literature  written  on  MAUT, 
or  consult  with  an  expert  practitioner  (the  latter  is  sug¬ 
gested  in  any  case) . 

B.  OVERVIEW 

The  most  typical  problem  setting  for  MAUT  applications 
arises  when  a  decision-maker  is  required  to  make  decisions 
among  two  or  more  alternatives,  whose  utility  varies  along 
several  dimensions  of  value.  As  pointed  out  by  Keeney  (MIT, 
1969,  p.  1€)  and  others,  in  reality  there  are  only  a  few 
decisions  which  are  based  on  just  one  measure  of  effective¬ 
ness.  This  realization  has  prompted  the  development  of  MAUT 
and  other  procedures  which  attempt  to  cope  with  more  complex 
problems.  Thus,  MAUT  is  essentially  a  descriptive  technique 
that  processes  information  according  to  specific  rules.  Data 
processing  is  accomplished  within  a  logical  but  flexible 
framework  founded  upon  quantitative  combinations  of  evidence. 
Evidence  is  brought  to  bear  on  the  outcomes  to  be  evaluated 


by  locating  them  on  various  dimensions  of  value.  The  located 
measures  are  then  aggregated  according  to  a  combination  rule 
which  weighs  the  relative  importance  of  each  dimension.  If 
the  model  is  successful,  it  will  present  the  outcome  or 
alternative  that  represents  the  greatest  worth  to  the  decision¬ 
maker  [Pacific  Missile  Test  Center,  1979,  p.  12].  The  major 
building  stones  of  MAUT  are: 

(1)  The  basic  structural  principle  in  MAUT  is  hierarchical 
decomposition.  This  means  that  the  evaluation  problem  is 
broken  down  or  decomposed  from  the  general  problem  to  specific 
components.  The  model  provides  the  structure  and  rules  neces¬ 
sary  to  investigate  and  integrate  the  interrelationships  of 
all  components . 

(2)  The  selected  value-dimensions  are  scaled,  and  utility 
functions  are  established. 

(3)  Global  utility  in  a  MAUT  procedure  is  expressed  by  a 
single  number.  Thus  multdimensional  outcomes  must  be  trans¬ 
formed  into  a  single  figure  of  merit  such  as  utility,  system 
worth  or  system  effectiveness.  This  is  done  by  means  of  some 
aggregation  rules. 

Finally,  according  to  Winterfeldt  and  Fischer  (1973,  p.  1) 
and  many  others,  MAUT  combines  a  class  of  psychological  measure¬ 
ment  that  can  be  applied  to  the  evaluation  of  alternatives 
with  multiple  value  relevant  attributes.  This  notion,  of 
MAUT's  capability  of  handling  psychological  measurement,  or 
in  other  words,  subjective  human  judgment,  is  paramount  in 
the  appreciation  of  the  method. 
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C.  DEFINITIONS  CLARIFICATION 


Before  the  discussion  proceeds  further  into  more* detailed 
presentation  of  MAUT,  it  is  necessary  to  provide  clarification 
of  terminology  that  is  used  interchangeably  in  the  literature 
and  in  this  thesis  as  well. 

The  various  names  and  acronyms  used  to  identify  MAUT  in 
general  have  been  already  discussed  in  the  Introduction  (see 
footnote  in  Chapter  I,  p.  14),  and  therefore  are  not  repeated 
here. 

The  type  of  decisions  which  are  discussed  in  this  thesis 
are  referred  to  as  multi-objective ,  multi-attribute  or  multi¬ 
criteria  decisions.  The  three  names  represent  three  points 
of  view  on  the  same  type  of  decision.  “Objective"  generally 
indicates  the  "direction"  in  which  one  should  strive  to  do 
better.  The  characteristic  phrasing  of  an  objective  can  be, 
for  example,  "maximize  air-to-air  effectiveness."  "Attribute" 
is  the  term  by  which  the  extent  of  achieving  the  objective 
is  measured  [Keeney  and  Raiffa,  1976,  p.  34].  Therefore, 
attribute  is  also  called  "measure  of  effectiveness"  (e.g., 
by  Keeney  and  Nair  (1977)).  The  attribute  for  the  above  objec¬ 
tive  can  be,  for  example,  "kill-ratio  in  air-to-air  engagements. 
An  attribute  may  be  based  on  purely  objective  scale,  or  partly 
subjective  scale  (objective  measurement  is  subjectively  evalu¬ 
ated)  ,  or  purely  subjective  scale  [Edwards,  1977,  p.  352]. 
"Criteria"  is  a  required  level  of  achievement.  For  example, 
one  may  establish  criteria  for  air-to-air  effectiveness  such 
as:  "ratios  of  1:1  is  poor,  1:5  is  fair  and  1:10  is  very 
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good."  Criteria  corresponds  to  "goal"  in  the  sense  that  goals 
clearly  identify  a  level  of  achievement  to  strive  toward. 

To  continue  the  above  pattern  of  examples,  the  goal  might 
be,  "strive  to  achieve  1:12  kill  ratio." 

The  decisions  are  multi-objective-attribute-criteria 
because  they  have  several,  sometimes  conflicting,  objectives 
(e.g.,  "maximize  air-to-air  effectiveness,”  "minimize  cost," 
"cause  minimal  environmental  disturbance  (such  as  noise  and 
smoke)").  These  objectives  have  corresponding  attri¬ 
butes  and  a  set  of  criteria  for  each  (which  are  in  fact  some 
discrete  consequences  on  a  utility  function  scale) . 

As  a  consequence  of  the  previous  discussion,  objectives , 
attributes  and  criteria,  although  distinguished  from  each 
other,  all  may  be  essentially  interpreted  as  areas  on  which 
an  evaluation  of  an  entity  is  based.  In  defining  these  areas 
of  evaluation  two  more  terms  are  in  use:  "value-dimensions" 
(e.g.,  Edwards  (1977))  and  "factors . "  Thus,  in  the  jargon, 
all  five  terms  are  used  interchangeably  to  describe  areas  of 
evaluation,  and  in  that  sense  they  are  considered  synonymous. 

As  indicated  by  all  decision  analysts,  objectives  are 
characterized  by  an  hierarchical  nature.  In  fact,  everyone 
who  has  seriously  thought  about  objectives  in  a  complex  prob¬ 
lem,  has  come  up  with  some  sort  of  hierarchy  of  objectives. 

The  lower-level  objectives  can  also  be  thought  of  as  the  means 
to  an  end,  the  end  being  the  higher-level  objective  [Keeney 
and  Raiffa,  1976,  p.  41].  The  hierarchical  construct  of  the 
objectives  is  also  called  a  "value- tree"  (e.g.,  Edwards  and 
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Newman  (1980) ).  A  value-tree  can  be  a  construct  of  objec¬ 
tives,  of  value-dimensions,  of  attributes,  of  criteria,  or 
of  factors.  (In  each  case,  of  course,  the  various  branches' 
definitions  are  differently  phrased.)  The  upper  level  objec¬ 
tives  (or  value-dimensions,  etc.,  of  the  hierarchical  con¬ 
struct  are  also  called  "categories"  (which  are  divided  into 
"subcategories")  or  "branches "  (of  the  "value-tree" ) .  There 
is  no  strict  definition  at  what  levels  these  notations  are 
valid.  Usually,  categories  are  used  to  describe  only  the 
highest  level  of  objectives  (only  below  the  top,  overall 
objective) .  Objectives  at  all  levels  below  are  referred  to  as 
subcategories.  Branches  usually  consist  of  objectives  at  all 
levels  except  the  bottom-level  ones.  Thus,  while  speaking 
about  subcategories  or  branches,  additional  indications  are 
required  to  know  exactly  at  what  level  a  specific  objective 
is. 

The  bottom-level  components  in  the  tree  are  called  also 
"twigs"  (e.g.,  Edward  and  Newman,  (1980)),  or  " entry - 1 eve Is" 
(e.g..  Decisions  and  Designs  (1980)).  In  MAUT,  as  will  be 
explained  in  more  detail  later  in  this  chapter,  the  actual 
evaluation  of  alternatives  is  done  only  at  the  entry-levels. 

The  weighted  utilities  at  any  higher  level  are  an  automatic 
outcome  of  "multiplication  through  the  tree."  Thus,  attri¬ 
butes,  i.e.,  evaluation  tools  are  required  only  for  the 
bottom-level  objectives.  Keeney  and  Raiffa  (1976)  are  con¬ 
sistent  in  their  definition  of  attribute,  and  thus  refer  to 
attributes  only  at  the  bottom-level  items,  while  all  the  others 
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at  higher  levels  are  called  objectives.  The  same  way  of 
notation  appears  in  Stillwell's  technical  report  [Social  Science 
Research  Institute,  1981]  value-trees  (see  Figure  IV- 1) . 

Edward  and  Newman  (1980)  are  less  strict  and  use  attribute 
as  synonymous  with  value-dimension,  without  identification 
of  the  level  at  which  they  are  located,  or  how  they  are 
phrased.  The  broad  meaning  attributed  by  Edwards  and  Newman 
to  the  term  "attribute"  is  reflected,  for  example,  in  the 
sentence  "stakehold  are  sources  of  value-attributes.  An 
attribute  is  something  that  the  stakeholders  care  about..." 
[Edwards  and  Newman,  1980,  Chapter  2,  p.  2] .  A  "proxy  attri¬ 
bute"  is  one  that  reflects  the  degree  to  which  an  associated 
objective  is  met,  but  does  not  directly  measure  the  attribute 
[Keeney  and  Raiffa,  1976,  p.  55].  Decisions  and  Designs  (1980, 
p.  12)  uses  the  term  surrogate  to  express  the  same  idea. 

Several  terms  are  used  for  the  numerical  evaluation  of  an 
alternative  over  a  specific  attribute.  These  are:  "measure 
location, "  "utility  location, "  "single  attribute  utility" 

(all  the  above  definitions  are  found  in  Edwards  and  Newman, 
(1980)).  Another  term  is  "scores"  (e.g.,  used  by  Donnell 
and  Ulvila  [Decisions  and  Designs,  1980]). 

"Weights "  are  called  also  "importance  weights,"  although 
the  latter  notation  is  rejected  by  Donnell  and  Ulvila  [Deci¬ 
sions  and  Designs,  1980,  p.  9]  with  the  argument  that  they  do 
not  really  express  importance.  Weights  appear  also  as 
"scaling  constants"  (e.g.,  in  Keeney  and  Nair  (1977)). 
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OVERALL  VALUE 


2nd  level 
weights: 


Figure  xv-x. 

_ _ r^e^TsT 


The  purpose  of  this  discussion  concerning  MAUT  terminology 


was  to  provide  a  background  for  subsequent  text.  The  reader 
should  be  aware  of  the  usage  of  some  synonymous  terms  inter¬ 
changeably  used  in  this  thesis. 

D.  THE  BASICS  OF  MAUT 

Let  Xj  be  a  set  of  possible  values  of  attribute  j  and 
be  a  particular  value  of  attribute  j.  As  an  example,  the  j*"*1 
attribute  could  be  the  color  of  a  car,  which  might  be  one  of 
several  attributes  one  might  consider  in  buying  a  car.  Let 
there  be  a  total  of  n  attributes  under  consideration  (e.g., 
price,  style,  economy,  etc.).  Any  one  alternatives  (e.g., 
car  types)  can  be  represented  as  a  particular  attribute 
combination: 


Lx1'x2' 


'V 


n 


} 


The  tendency  to  prefer  one  alternative  over  another  can  be 
represented  by  a  construct  called  "utility"  and  denoted  U. 

The  utility  of  an  alternative  is  a  function  of  the  particular 
values  of  the  specific  alternative  over  all  attributes  under 
consideration : 


ui  =  F(x1,x2, . . . ,Xj , . . . ,xn) 

where  U^  is  the  overall  utility  of  the  i^1  alternative.  Most 
authors  dichotomize  the  functions  as  being  additive  or  multi¬ 
plicative.  The  simplest  and  most  prevailing  form  of  additive 
models  is  the  linear  additive  model: 


U  =  wi*u(x1)  +  w2*u(x2)  +  ...  +  wn-u(xn) 


where : 


w.  is  the  normalized  importance  weight  of  the  j 
3  n 

attribute  (such  as  £  w.  =  100  or  1,  as  preferred) 

j-1  3 

u(Xj)  is  the  normalized  utility  function  (usually  on 
a  scale  of  0-10,  0-100,  or  0-1000)  on  the 
individual  attribute. 


Let  =  u(Xj),  then 


U.  =  l  w.  u.  . 


3=1  3  U 


where: 


-  Uij 


-  U, 


x.  k. 

is  the  utility  of  the  i  alternative  over  the 
single  jth  attribute; 

is  the  overall  utility  of  the  i*"*1  alternative. 
[Edwards,  1977,  p.  253] 


Other  types  of  additive  models  are  those  which  include  inter¬ 
actions  [Pacific  Missile  Test  Center,  1979,  p.  A-6] .  Those 
will  not  be  discussed  here. 

The  multiplicative  models  are  generally  of  the  form 


W1  w2  wn 

U  =  wnx.  .x,  ...x  [Pacific  Missile  Test 

u  1  1  n  Center,  1979,  p.  A-6] 


Or  in  the  form 
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1  +  kU (X) 


n 


I 

j-1 


[1  +  k  k.  u.  (x.)  ] 
JJJ 


where : 

is  a  multi-attribute  utili  function, 
are  constants  with  k  >  -1  and  0  <  k^  <  1,  and 
is  the  utility  function  of  the  individual 

attribute.  [Giaque,  1972,  p.  IV-19] . 

In  selecting  among  the  various  aggregation  methods  the  authors 
have  taken  several  approaches.  One  way  of  evaluating  a  model 
is  to  ask  whether  or  not  the  model  logically  reflects  the 
decision-making  process  in  a  valid  and  predictive  sense,  while 
the  second  way  is  to  ask  if  the  decision-making  process  can  be 
approximated  by  a  relatively  simple  model.  The  additive  ap¬ 
proach  is  less  compatible  with  the  value  independence1  assump¬ 
tion  that  is  assumed  by  both  additive  and  multiplicative 
versions.  Consequently,  Keeney  (1974)  has  proposed  use  of  a 
multiplicative  rather  than  additive  aggregation  rule.  But 
Edwards  (1977),  who  consistently  strove  for  model  simplicity, 
indicates : 

...in  the  presence  of  even  modest  amounts  of  measure¬ 
ment  error,  quite  substantial  deviations  from  value 
independence  will  make  a  little  difference  to  the 
ultimate  number  U.  (the  aggregate  utility)  and  even 
less  to  the  rank  Ordering  of  the  U.  values. 


U(X) 

k,  k . 
: 


^Roughly,  value  independence  means  that  the  extent  of  one's 
preference  for  location  (score)  x-  over  location  x.  of  specific 
attribute,  say,  X. ,  unaffected  by^the  position  of  the  entity 
being  evaluated  on  dimensions  X, ,X,,...,X  ,  and  so  on 
[Edwards,  1977,  p.  255].  *  J  n 
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Edwards  concludes : 


...the  additive  approximation  will  almost  always  work 
well  if,  for  each  dimension,  either  more  is  prefer¬ 
able  to  less  or  less  is  preferable  to  more  through 
the  range  of  the  dimensions  that  is  involved  in  the 
evaluation  for  all  available  values  of  the  other 
dimensions . 

Thus,  Edwards  and  others  conclude  that  additive  models  serve 
as  good  approximations  in  most  cases  and  are  the  most  appro¬ 
priate  in  terms  of  practical  usage  [Pacific  Missile  Test  Center, 
p.  A- 7] .  But  Edwards  admits  that  when  the  assumption  of  value 
independence  is  unacceptable  even  as  an  approximation,  much 
more  complicated  models  and  elicitation  procedures  that  take 
value  dependence  into  account  should  be  used,  like  those  pre¬ 
sented  by  Keeney  and  Raiffa  (1976).  Edwards  (1977,  p.  250) 
ends  up  with  the  following  conclusion: 

...theory,  simulation  computation  and  experience, 
all  suggest  that  weighted  linear  averages  yield 
extremely  close  approximations  to  very  much  more 
complicated  non-linear  and  interactive  ’true' 
utility  functions,  while  remaining  far  easier  to 
elicit  and  understand. 

(See,  for  example,  Dawes  and  Corrigan  (1974),  and  Einhorn 
and  Magarth  (1975).) 

Most  decisions  involve  uncertainties.  These  decisions 
are  referred  to  as  "risky  decisions."  Essentially,  the  conse¬ 
quence  is  using  expected  utilities  rather  than  simple  utili¬ 
ties.  Edwards  (1981)  emphasizes  that  the  expected  values 
should  be  calculated  on  the  natural  values  (i.e.,  performance 
measures)  of  the  attributes,  rather  than  on  the  normalized 
utilities  resulting  from  them.  When  the  probability  function 
for  the  various  possible  outcomes  is  known,  it  should  be  used. 

i 
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Otherwise,  probabilities  have  to  be  estimated,  and  then  utili¬ 
ties  become  subjective  expected  utilities.  There  is  a 
variety  of  ways  of  how  to  elicit  those  subjective  estimated 
probabilities.  It  starts  from  an  extreme  approach  such  as 
Bauer  and  Wegener's  (1977,  pp.  342-343)  who  found  out  (in  a 
specific  case)  that  probabilities  are  so  difficult  to  estimate 
successfully,  that  it  implies  an  assumption  of  equal  certainty 
or  uncertainty,  for  all  models  output,  and  tb  .s  completely  ig¬ 
nore  the  notion  of  risk.  At  the  other  extreme  stands  the 
sophisticated  assessment  of  subjective  probability  estimates 
of  utility,  introduced  by  Raiffa  (1968)  and  Keeney  and  Raiffa 
(1976).  Besides  the  expected  utility  notion,  uncertainties 
may  be  taken  care  of  by  sensitivity  analysis,  in  which  effects 
on  the  ultimate  outcome  resulting  from  deviations  from  the 
expected  values  are  tested  (Fisher,  1975,  p.  74] .  Another  ap¬ 
proach  is  using  special  value-dimensions  for  "risks  involved" 
as  done,  for  example,  by  Donnell  and  Ulvila  in  their  ASH 
model  [Decisions  and  Designs,  1980].  According  to  Edwards 
(1981),  "this  is  theoretically  inelegant,  but  definitely 
eligible. " 

E.  EDWARDS'  7  STEPS  OF  MAUT  PROCEDURE 
1.  General 

There  are  many  versions  of  MAUT  in  existence.  While 
all  refer  to  the  same  basic  ideas,  the  details  of  implemen¬ 
tation  differ  from  one  another.  Edwards  and  Newmann  (1980) 
introduce  a  7-step  MAUT  procedure,  which  is  a  development  of 
similar  procedures  introduced  by  Edwards  in  his  older 


publications  [Edwards,  1971,  1977].  Edwards'  approach  has 
been  selected  for  further  discussion  in  this  thesis  because 
of  the  following  reasons: 

(1)  Its  framework  applies  almost  to  any  MAUT  procedure. 

(2)  It  meets  all  six  very  important  characteristics  which 
Little  {1970,  pp.  466-485)  defined  as  vital  for  a  model  to  be 
useful.  According  to  Little  a  model  should  be: 

(a)  Simple  and  easy  to  understand  (though  not  simplistic) . 

(b)  Robust:  a  user  should  find  it  hard  to  make  the  model 
give  bad  answers. 

(c)  Easy  to  control:  a  user  should  be  able  to  make  the 
model  behave  the  way  she  or  he  wants  it  to  behave. 

(d)  Adaptive:  It  should  be  easy  to  update  in  terms  of 
both  parameters  and  structure  as  new  information  becomes 
available. 

(e)  Complete  on  important  issues:  in  conjunction  with 
simplicity,  this  implies  an  optimal  level  of  detail  and 
complexity,  which  balances  precision  with  scope.  (According 
to  Little,  "An  important  aid  to  completeness  is  the  incorpora¬ 
tion  of  subjective  judgments.") 

(f)  Easy  to  communicate  with:  a  user  needs  to  be  able  to 
change  inputs  simply,  and  obtain  outputs  quickly. 


2.  The  7  Steps1 

a.  Step  1 

Identify  the  objects  of  the  evaluation,  and  the 
function  or  functions  that  the  evaluation  is  intended  to  per¬ 
form.  Determine  the  relevant  alternatives  (or  options)  to 
be  evaluated.  They  can  be  defined  as  outcomes  of  some  actions, 
or  as  the  actions  themselves. 

b.  Step  2 

Identify  the  stakeholders.  A  stakeholder  is 
simply  an  individual,  group,  or  organization  with  a  reason  to 
care  about  the  decision,  and  with  enough  impact  on  the  decision¬ 
maker  so  that  the  reason  should  be  taken  seriously.  The  deci¬ 
sion-maker  is  usually  a  stakeholder  himself  and  not  just  a 
"judge."  So  he  (or  they)  must  be  equally  addressed  at  this 
step  like  any  other  stakeholder. 

c.  Step  3 

Elicit  from  the  stakeholders  (or  their  representa¬ 
tives)  the  relevant  value-dimensions  and  (often)  organize  them 
into  a  hierarchical  structure  called  a  "value-tree."  Develop¬ 
ment  of  a  value-tree  is  a  phase  of  interaction  between  the 
analysis  team  and  the  stakeholders.  Structuring  of  the  value- 
tree  is  done  in  a  top-to-bottom  order.  Judgment  must  be  used 
to  decide  where  to  stop  formalizing  the  hierarchy  by  considering 

^The  introduction  of  the  7  steps  are  based  mainly  on  three 
references:  Edwards  (1977),  Edwards  and  Newman  (1980),  and 
Edwards  (1981) .  Although  these  underly  the  whole  section,  they 
will  be  specifically  referred  to  where  necessary  within  the 
section.  In  some  cases,  the  discussion  expands  beyond  the 
scope  of  Edwards'  7  steps.  In  such  cases,  additional  references 
will  be  cited. 
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the  advantages  and  disadvantages  of  further  specification. 

The  more  an  objective  hierarchy  is  subdivided,  the  easier  it 
usually  is  to  identify  attribute-scales  that  can  be  objectively 
assessed.  But  of  course,  there  are  attributes  which  cannot 
be  objectively  measured  at  all.  Moreover,  by  going  down  too 
far,  not  only  the  simplicity  principle  is  violated,  but  the 
efficient  tool  of  human  judgment,  which  can  ’’leap  frog"  many 
steps  of  elaborate  "objective"  evaluations  is  not  fully  uti¬ 
lized.  (With  this  repect,  recall  Little's  comment,  saying 
that  in  the  attempt  to  reach  completeness  and  still  keep  the 
balance  between  simplicity  and  complexity,  incorporation  of 
subjective  judgment  is  an  important  aid) . 

In  many  practical  cases  it  is  hard,  if  not  impossi¬ 
ble,  to  reach  objectively  measured  attributes.  For  example, 
Donnell  and  Ulvila's  [Decisions  and  Designs,  1980,  pp.  31-32) 
model  of  the  Scout  Helicopter  evaluation,  although  relatively 
elaborate,  still  ends  up  with  most  of  the  bottom-level  attri¬ 
butes  that  are  not  readily  quantifiable  on  an  underlying  scale. 

A  means  to  reduce  the  number  of  attributes  is  mainly 
to  combine  some  of  them,  what  is  in  fact  equivalent  to  stop 
formalizing  at  a  higher  level  of  the  attributes  hierarchy,  or 
to  eliminate  the  unimportant  ones.  Here  one  can  use  the 
"test  of  importance,"  introduced  by  Ellis  (1970) ,  in  which 
the  decision-maker  is  asked  whether  he  feels  the  best  course 
of  action  could  be  altered  if  a  specified  objective  was  ex¬ 
cluded.  An  affirmative  response  would  obviously  imply  that 
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the  objective  should  be  included.  Edwards  (1981)  mentions 
that  it  is  desirable  to  get  the  stakeholders'  common  acceptance 
of  the  value-tree,  although,  he  indicates,  "you  can  never 


satisfy  them  all." 

Donnel  and  Ulvila  [Decisions  and  Designs,  1980, 
pp.  7-8]  count  the  important  benefits  the  hierarchical  decom¬ 
position  provides  as  follows: 

(1)  It  breaks  the  elicitation  process  up  into  "chunks"  of 
manageable  size. 

(2)  It  organizes  the  presentation  of  the  final  results, 
highlighting  the  most  important  factors  without  losing  the 
ability  to  retrieve  details  when  desired. 

(3)  It  limits  the  required  assessments  to  comparison 
among  attributes  that  are  closely  related  in  meaning,  and 
therefore  relatively  easy  to  weigh  against  one  another. 

d.  Step  4 

For  each  stakeholder  group  assess  the  relative 
importance  of  each  of  the  value-dimensions.  Since  values 
change  over  time,  the  weights  should  be  re-elicited  in  situa¬ 
tions  in  which  the  program  is  periodically  re-evaluted. 

As  emphasized  by  Helm  [Pacific  Missile  Test  Center,  1979,  p.  12] 
"MAUT  is  especially  appropriate  for  the  applied  setting,  since 
in  addition  to  its  flexible  structure  it  incorporates  inter¬ 
active  methodology  for  data  manipulation."  Consequently ,  as 
new  data  are  gathered,  existing  information  is  revised  in  the 
light  of  the  new  data.  This  process  can  be  repeated  as  often 
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as  required.  That  feature  protects  MAUT  from  the  fate  of 
many  static  models,  which  quickly  reach  obsolescence. 

Step  4  is  the  second  of  the  only  two  steps  in 
which-  according  to  Edwards,  stakeholders  are  involved.  (The 
first  one  is,  as  recalled,  the  value-dimensions  elicitation.) 
Obviously,  it  is  desired  and  in  most  cases  even  possible  to 
reach  a  common  stakeholders '  agreement  to  both  the  value-t;  **- 
construct  and  the  weights.  The  recommended  way  to  reach  such 
agreement  is  through  a  face-to-face  open  discussion.  If  agree¬ 
ment  is  not  reached,  the  ultimate  decision-maker  may  judge 
and  decide  about  the  final  value-tree  construct  and  weights. 
Another  option  is  to  use  more  than  one  set  of  weights  (although 
the  value-trees  must  be  identical)  for  parallel  comparative 
evaluations.  According  to  Edwards  (1981)  in  many  practical 
cases,  the  ultimate  outcome  of  different  weight  sets  was 
selection  of  the  same  alternative.  That  is  explained  by  the 
fact  that  in  practical  cases,  weight  sets  which  are  determined 
by  reasonable  people,  usually  only  differ  moderately,  thus 
the  dominant  discriminator  is  score  rather  than  weight. 

Edwards  and  Newman  (1980)  prefer  the  weighting  method  called 
"ratio  weighting."  The  procedure  for  this  method  is  as  follows 

(1)  Place  the  attributes  in  rank  order  of  importance. 

(2)  Assign  a  value  of  10  to  the  least  important  attribute. 

(3)  Assign  to  the  next  attribute  a  value  according  to  your 
evaluation  of  how  much  it  is  more  important  than  the  least 
important  one.  For  example,  if  it  is  twice  as  important  as 
the  least  important  attribute,  it  should  be  assigned  the  value 
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of  20 .  Go  up  the  rank  order  and  assign  relative  values  to 
all  attributes.  Ties  are  permitted.  Normalize  the  values 
assigned  to  a  scale  of  0-1,  0-100,  or  as  desired.  This  is 
done  by  summing  up  all  important  values,  and  dividing  each 
individual  value  by  the  sum.  The  result  is  the  attribute's 
importance  weight  on  a  sacle  of  0-1.  For  0-100  scale,  multi¬ 
ply  by  100.  When  you  have  a  value-tree,  use  it  to  ease  the 
weighting:  instead  of  generating  weights  for  all  attributes 

together,  do  the  same  procedure  for  each  level  separately, 
and  within  the  levels — for  each  group  of  attributes  (which 
are  under  a  common  upper-level  attribute)  separately.  Start 
the  procedure  from  the  top  and  proceed  to  the  bottom.  The 
final  weight  of  each  attribute  is  the  multiplication  of  his 
original  weight  by  the  weights  of  all  attributes  which  are 
above  it,  in  its  branched  pattern  up  to  the  top.  (A  detailed 
example  can  be  found  in  Edwards  and  Newman  (1980,  Chapter  4, 
pp.  12-14)  .  This  is  called  "multiplying  through  the  tree." 

While  Edwards  and  Newman  (1980,  Chapter  4)  pre¬ 
sent  some  additional  weighting  methods  which  are  even  simpler 
than  the  ratio  weighting,  there  are  also  more  complicated 
methods  (see  for  example  Keeney  and  Nair  (  1977,  pp.  310-312)  ), 
which  are  essentially  based  on  exploring  trade-off  ratios  of 
scores  among  the  various  attributes, 
e.  Step  5 

Ascertain  how  well  each  object  of  evaluation 
serves  each  value-dimension  at  the  lowest  level  of  the  value- 
tree.  This  is  expressed  in  terms  of  location-measures  (or 
single  attribute  utilities,  or  scores)  .  Those  numbers  can  be 

83 


i 


purely  subjective  or  some  sort  of  transformation  on  objective 
measures.  Edwards  (1977,  p.  252)  distinguishes  between  purely 
subjective  value-dimension,  partly-subjective — in  which  the 
units  of  measurement  are  objective  but  the  location-measures 
are  subjectively  evaluated,  and  purely-ob j ective--in  which 
location-measure  can  be  measured  unjudgmentally  in  objective 
units  before  the  decision. 

An  important  part  of  the  location  measuring  pro¬ 
cedure  is  to  construct  a  utility  function  in  which  the 
location-measure  of  each  alternative  can  be  evaluated.  While 
some  of  the  users  prefer  a  sophisticated  development  of 
dimension-by-dimension  utility  curves  (like  Keeney  and  Raiffa 
(1976)),  Edwards,  who  consistently  strives  for  simplicity,  is 
a  great  advocate  of  using  linear  ("straight  line")  utility 
functions.  According  to  Edwards  (1981),  in  most  cases  prefer¬ 
ences  are  described  basically  by  a  straight  line,  or  by  a 
bi-linear  function  with  only  one  interior  maximum  (or  minimum) . 
Edwards  and  Newman  (1980,  Chapter  5,  p.  8)  point  out  that 
the  use  of  linear  and  bi-linear  location  measures  is  an  enor¬ 
mous  simplification,  "which  is  very  much  out  of  the  traditional 
spirit  of  formal  decision  analysis."  But  they  emphasize  that 
"if  one's  desirability  or  utility  function  increases  steadily, 
or  decreases  steadily,  or  has  one  interior  maximum,  then  this 
approximation  will  work  so  well  that  there  is  little  point 
in  using  anything  more  sophisticated."  Donnell  and  Ulvila 
[Decisions  and  Designs,  1980,  pp.  10-21]  distinguish  among 
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three  methods  of  scaling:  relative  scales,  ratio  scales  and 
absolute  scales.  All  three  methods  assume  linearity.  The 
absolute  method  requires  determination  of  range  of  performance 
that  starts  at  some  very  low  "true  zero"  level,  and  ends  at 
some  theoretical  maximum  level  of  performance.  The  advantage 
of  this  method  is  that  it  indicates  not  only  the  relative 
preference  order  between  alternatives,  but  it  also  indicates 
to  what  extent  an  alternative  is  "good"  or  "bad"  relative  to 
some  acceptable  absolute  reference  scale.  The  disadvantages 
are  that  it  is  hard  to  define  the  scaling  range,  and  because 
the  true  zero  is  so  low,  the  spread  in  the  alternatives' 
scores  is  relatively  small  and  hard  to  distinguish.  In  the 
ratio  scale  the  analyst  can  avoid  the  problem  of  defining 
"perfect"  capability,  and  the  best  alternative  on  each  dimen¬ 
sion  serves  as  the  upper  limit  of  the  range.  Other  alterna¬ 
tives  are  evaluated  relative  to  the  best.  But  this  method 
still  requires  defining  an  "unacceptable"  point  of  zero  value 
as  in  the  absolute  scale  method.  The  relative  scale  does  not 
require  definition  of  end  points  at  all.  The  best  alternative 
at  each  dimension  is  used  as  the  upper  limit  of  the  scale,  and 
the  worst  as  the  bottom  limit.  Other  alternative  evaluations 
are  located  in-between  according  to  their  estimated  (or  objec¬ 
tively  measured)  location  relative  to  both  limits.  The  main 
advantage  of  this  method  is  its  simplicity  and  its  "modest" 
information  requirements.  The  disadvantage  is  that  it  does 
not  indicate  whether  an  alternative  is  "good”  or  "bad, "  only 
"better"  or  "worse"  are  meaningful.  All  three  methods,  if 
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properly  implemented,  provide  the  same  rank  order  of  alterna¬ 
tives  and  even  the  same  ratio  of  preferences  among  them.  In 
most  decisions  relative  order  is  sufficient,  and  if  there  is 
no  specific  need  for  evaluation  in  absolute  terms,  according 
to  Donnell  and  Ulvila  [Decisions  and  Designs,  1980,  p.  7]  the 
relative  scaling  method,  whose  advantages  outweigh  its  problems, 
is  preferred. 

Edwards  (1981)  views  the  scoring  process  as  done 
essentially  by  experts.  In  any  case,  he  recommends  to  avoid 
letting  stakeholders  do  the  scoring  because  that  is  a  poten¬ 
tial  source  of  biased  outcomes, 
f.  Step  6 

Aggregate  the  location-measures  with  the  impor¬ 
tance  weights  into  an  overall  utility  for  each  alternative. 

As  mentioned  before,  Edwards,  and  many  others 
prefer  the  additive  aggregation  rule  to  the  multiplicative 
one . 

Another  issue  that  may  be  recalled  is  execution 
of  several  aggregations  (but  not  too  many!)  using  different 
sets  of  weights.  Those  can  be  presented  concurrently  for 
decision-maker's  judgment.  Such  presentations  may  convey 
to  him  in  a  visible  manner,  controversies  among  groups  of 
stakeholders,  if  they  are  really  significant  [the  word  signi¬ 
ficant  refers  to  both  controversies  and  stakeholders.  To 
be  worth  separate  aggregations  both  should  be  significant. 

R.G.] . 
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An  issue  which  has  a  great  practical  significance 
that  is  suggested  by  Edwards  and  Newman  (1980,  Chapter  6, 
pp.  11-12)  is  subaggregation.  As  they  say,  aggregation  need 
not  be  an  all-or-nothing  affair.  If  value-trees  have  been 
developed,  one  can  select  an  appropriate  level  of  higher  order 
value,  and  aggregate  up  to  it.  Then  MACJT  scores  on  each 
branch  separately  can  be  presented  as  a  "value  profile" — an 
aggregate  but  still  informative  summary  of  how  objective  evalu 
ation  stands  with  respect  to  each  of  the  higher-level  values 
considered  relevant  to  its  assessment.  That  gives  information 
at  whatever  level  of  detail  seems  to  be  just  right  for  the 
purpose  on  hand. 

g.  Step  7 

Perform  sensitivity  analysis.  As  true  for  any 
sensitivity  analysis,  the  underlying  question  at  this  step 
is  whether  a  change  in  the  various  inputs  or  input-evaluations 
will  lead  to  a  different  conclusions.  According  to  Edwards 
and  Newman  (1980,  Chapter  7,  p.  1),  the  most  important  kind 
of  sensitivity  to  look  at  is  sensitive  to  weigths.  This  is 
important  both  because  weights  are  the  essence  of  value 
judgment,  and  because  weights,  being  purely  subjective  num¬ 
bers  about  which  people  disagree,  are  more  likely  to  be  in 
dispute  than  location  measures,  which  may  be  objective,  may 
depend  on  the  judgment  of  experts — or  may  be  in  some  cases 
also  matters  of  intense  controversy.  If  there  is  some  debate 
about  whether  a  branch  or  an  attribute  belongs  in  the  analysis 


at  all,  it  can  be  in  effect  eliminated  in  the  sensitivity 
analysis  by  giving  it  a  weight  of  zero,  or  almost  zero. 

F.  CONCLUSIONS 

This  chapter  attempted  to  provide  the  reader  a  basic 
acquaintance  with  MAUT.  As  noted,  MAUT  has  variety  of  ver¬ 
sions.  In  this  context,  the  simpler  ones  were  addressed, 
with  several  references  to  the  more  complex  ones.  But  it 
should  be  emphasized  that  the  simplest  versions  of  MAUT, 
like  the  one  introduced  by  Edwards  and  Newman  (198C)  ,  may 
not  be  adequate  to  handle  the  more  complex  acquisition 
decisions.  Edwards  himself  admits  that  he  has  not  had  experi¬ 
ence  with  such  complex  problems  {Edwards,  19  81]  .  For  example, 
Donnell  and  Ulvila's  [Decisions  and  Designs,  1980]  value-tree 
used  for  the  Advanced  Scout  Helicopter  evaluation  is  much 
more  complex  than  the  "trees"  exhibited  in  Edwards'  examples. 
On  the  other  hand,  Donnell  and  Ulvila’s  linear  utility  func¬ 
tion  assumption,  their  usage  of  relative  scale,  their  prefer¬ 
ence  for  additive  linear  aggregation  rule,  or  the  way  they 
treat  uncertainties,  corresponds  more  to  Edwards'  approaches, 
than  to  the  more  sophisticated  ones  suggested,  for  example^  by 
Raiffa  (1968),  Keeney  and  Raiffa  (1976),  or  Keeney  and  Nair 
(1977).  In  any  case,  the  selection  of  the  approach  depends 
a  great  deal  on  the  object  of  evaluation  and  on  the  prefer¬ 
ences  of  the  decision-maker.  Nevertheless,  many  approaches 
and  techniques— the  simple  and  the  more  complex — are  available 
for  use  as  required.  The  conflict  between  the  simplicity 
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desired  and  the  complexity  required  for  acquisitio. 
will  be  addressed  in  the  next  chapter. 
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V.  IMPLEMENTATION  OF  MAUT  FOR  TOP-LEVEL  ACQUISITION 

DECISION-MAKING 


A .  GENERAL 

The  purpose  of  this  chapter  is  to  highlight  some  of  the 
implementation-related  issues,  in  applying  MAUT  to  top-level 
acquisition  decision-making.  No  doubt,  acquisition  decisions 
are  of  the  most  complex  kind.  Although  MAUT  was  originally 
geared  to  complex  decisions,  there  are  still  some  important 
problems,  stemming  mainly  from  the  complexity  of  the  acquisi¬ 
tion  decisions.  These  problems  should  be  resolved  in  order 
to  make  it  more  likely  that  MAUT  be  used  in  acquisition  decision¬ 
making.  Many  of  the  implementation  problems  may  be  associated 
with  each  individual  decision.  Those,  of  course,  cannot  be 
addressed  in  this  context.  Only  the  issues  which  are  common 
to  most  cases  will  be  mentioned  here. 

Neither  the  thesis  in  general,  and  this  chapter  in  particular, 
pretend  to  cover  all  implementation  aspects,  nor  are  they  supposed 
to  be  a  complete  "user's  manual."  Thus,  a  preliminary  recom¬ 
mendation  is  that  the  user  should  be  helped  and  consulted  by 
MAUT's  skilled  field  practitioners.  According  to  Edwards  (1981) , 
although  there  are  many  who  deal  with  MAUT,  "there  are  only  a 
few  real  skilled  practitioners  around."  This  is  by  no  means 
to  say  that  those  practitioners  will  do  the  evaluation  by 
themselves.  It  is  to  say  that  their  help  is  vital.  Basically, 
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the  MAUT  procedure1  is  managed  at  staff  level  by  a  team  of 
MAUT's  experienced  practitioners,  usually  externally  hired, 
and  combined  with  staff  members.  (The  combined  group,  here¬ 
after,  will  be  called  "the  analysis  team") .  The  analysis 
team  works  in  close  interactions  with  the  various  stakeholders, 
experts,  and,  of  course,  the  decision-maker (s) .  Those  inter¬ 
actions,  which  are  paramount  for  MAUT  procedure  scuccess,  will 
be  discussed  in  more  detail  later  in  this  chapter. 

A  final  comment,  which  is  preliminary  to  any  further  dis¬ 
cussion  of  the  implementation  issues,  is  that  MAUT  procedure 
must  be  acceptable  to  the  final  decision-maker.  This  notion 
has  been  raised  by  several  of  the  people  involved,  by  means 
of  the  questionnaire  and  the  interviews  (e.g. ,  Connolly  (1981) 
and  also  see  Appendix  A,  p.  227).  Furthermore,  one  of  the 
questionnaire  respondents  emphasizes  the  potential  contribu¬ 
tion  the  decision-maker's  experience  and  intuition  may  offer 
to  models  by  steering  them  from  the  early  stages.  According 
to  this  commentator,  "'Good'  decision-making  is  enhanced  by 
bringing  them  [the  models,  R.G.]  in  early,  so  that  data  base, 
etc.,  is  organized  to  their  [the  decision-makers',  R.G.] 
'taste'"  [Appendix  A,  p.  227]. 


The  words  "procedure,"  "technique,"  and  "model"  are  used 
interchangeably  in  this  chapter  to  refer  to  the  implementation 
form  of  MAUT.  There  are  practitioners  such  as  Edwards  (1981) 
who  have  reservations  about  using  the  term  "model"  in  this  con¬ 
text,  since  model  is  some  representation  of  real  phenomena, 
and  MAUT  is  merely  an  analysis  and  communication  mechanism. 
("Algebra  is  also  not  a  model.")  But  in  the  prevailing  jargon 
the  MAUT  framework  for  specific  analysis  is  called  a  "model" 
(see  for  example  Decisions  and  Designs  (ISHTO)  .  Thus,  this 
term  will  be  used  in  that  sense  in  this  chapter  as  well. 
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The  various  implementation  issues  will  be  addressed 
by  their  order  of  arrival  according  to  KAUT's  7  steps. 


B.  IMPLEMENTATION  ISSUES 
1.  Step  1 

As  recalled,  at  this  step  the  scope  of  the  decision 

is  determined  and  alternatives  defined.  That  brings  about 

the  question:  what  type  of  acquisition  decisions  is  the  MAUT 

procedure  applicable?  Edwards  (1977,  p.  248)  says: 

I  don't  understand  the  differences  among  evaluations 
of  plans,  evaluations  of  ongoing  projects  and  evalua¬ 
tions  of  complete  projects;  all  seem  to  me  to  be 
instances  of  the  same  kind  of  intellectual  activity. 

MAUT  can,  and  I  believe,  should  be  applied  to  all 
three;  the  only  difference  is  that  in  ongoing  or 
complete  projects  there  are  more  opportunities  to 
replace  judgmental  estimates  of  locations  on  value 
dimensions  with  utility  transforms  on  actual 
measurement — still  subjective,  but  with  firmer 
ground  in  evidence. 

A  similar  attitude  is  taken  by  Hermann  (1981)  who  says: 

I  am  not  sure  that  there  is  such  a  clear  distinc¬ 
tion  between  source  selection  decision  and  status 
review  decision  session  [i.e..  Milestone  session, 

R.G. ] . 

In  fact,  there  should  not  be  a  clear  distinction.  As  men¬ 
tioned  in  Chapter  II,  Milestones  I  and  II  are  designated 
to  approve  the  alternative (s)  and  authorize  the  acquisition 
process  to  proceed  into  the  next  phase.  Apparently,  the 
approval  of  the  alternatives  is  based  on  their  status  review. 
But  from  another  aspect,  each  alternative  system  is  usually 
identified  with  a  specific  contractor.  So  that  is  in  essence 
the  linkage  between  the  system/contractor  Source  Selection 
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and  the  Milestones  decisions.  The  above  mentioned  notion 
contradicts  some  people's  arguments  (such  as  Trimble  (15*81)) 
that  the  DSARC  is  not  supposed  to  make  any  kind  of  comparison 
between  alternative  systems  since  that  is  exclusively  the 
Services'  business.  According  to  Trimble,  DSARC 's  basic  job 
is  to  recommend  to  approve  or  disapprove  the  Service  selec¬ 
tion.  Thus,  with  this  pattern  of  logic  Trimble  concludes  that 
MAUT  procedure  might  be  most  useful  at  the  Department  level, 
and  especially  for  Source  Selection  (which,  in  fact,  already 
uses  such  procedures).  One  of  the  questionnaire  respondents, 
a  program  manager,  concludes  his  lengthy  written  analysis 
with  four  conclusions  which  essentially  match  Trimble's  per¬ 
ception.  These  are: 

(1)  Models  are  good  tools  for  evaluating  two 
basically  'technically  equal'  systems,  i.e.,  Source 
Selection. 

(2)  Models  are  useful  for  prototype  evaluation. 

(3)  They  are  o.c  limited  use  in  the  A-109  proc¬ 
ess,  even  when  comparing  similar  systems. 

(4)  They  are  of  no  use  in  unlike  alternatives 
(e.g.,  TOW  missile  vs.  A-10  aircraft,  to  kill 
tanks)  . 

While  it  absolutely  agrees  with  the  first  two  conclu¬ 
sions,  the  thesis  analysis  disagrees  with  the  last  two.  In 
response  to  the  third  conclusion,  as  explained  before,  it 
argues  that  there  should  not  be  a  difference  in  essence 
between  "  drc  A-109  process"  (or  "DSARC  process,'*  or  any 
similar  new  process)  decisions,  and  top  departments  decisions, 
such  as  Source  Selection  for  major  systems.  If,  in  fact. 
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there  is  a  difference,  it  steins  partly  from  weakness  of 
evaluation  capability  of  the  DSARC  process  decision  bodies, 
rather  than  from  principle.  Secondly,  the  flexible  nature  of 
MAUT  may  allow  its  use  in  assisting  top-level  decisions,  even 
when  they  change  shape  while  moving  up  the  ladder  of  the 
"DSARC  Process"  hierarchy.  The  fourth  conclusion  raises  a 
set  of  questions  with  respect  to  how  to  compare  unlike  alterna¬ 
tives  or  old  or  with  new  ones.  According  to  Edwards  (1981) , 
it  is  harder,  because  the  range  of  value  dimensions  to  be 
considered  is  greater,  but  it  is  possible.  "In  fact,"  he 
says,  "we  were  able  to  generate  value  trees  [to  such  evalua¬ 
tions,  R.G.]  never-the-less . "  In  the  case  of  unlike  alterna¬ 
tives  there  might  be  many  low  level  attributes  which  are  not 
common  to  all  alternatives.  On  the  other  hand,  since  all 
alternatives  aim  to  achieve  common  upper-levels  objectives, 
the  upper  levels  value  dimensions  are  probably  shared  by  all. 
Thus,  a  value  tree  may  consist  of  bottom  level  attributes 
accumulated  from  all  alternatives,  while  each  alternative  is 
actually  evaluated  only  on  those  attributes  which  are  rele¬ 
vant  to  it,  and  is  scored  0  on  the  irrelevant  ones. 

A  more  concise  option  is  to  construct  different  value 
trees  for  each  alternative,  which  includes  for  each  alternative, 
just  the  relevant  attributes,  and  thus  may  differ  at  the 
entry-levels,  but  be  identical  at  the  upper  levels.  Old  vs. 
new  alternatives  differ  mainly  in  to  what  extent  objectively 
measured  attributes  are  used,  and  in  the  degree  of  uncertain¬ 
ties.  In  such  a  case  it  is  possible  to  use  a  specific 
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value-dimension  subjective  scale  for  on-?  alternative  (proba¬ 
bly  the  new,  yet  "unknown"  one) ,  and  more  objectively-based 
scale,  for  another  (probably  for  the  old,  well-known  one) . 

Uncertainties  are  taken  care  of  by  expected  value 
computation  (or  estimation)  and  sometimes  by  applying  special 
value  dimensions  of  risk  (see  more  detail — this  chapter,  p.  110). 
Thus  there  is  no  fundamental  problem  in  applying  MAUT  to  "old 
vs.  new"  evaluation  context.  Some  examples  of  MAUT  actual 
implementation  for  "old  vs.  new"  evaluations  were  mentioned 
in  Chapter  III  (see  p.  63)  such  as:  Airport  for  Mexico  City, 
Energy  supply  for  Southern  California,  or  Advanced  Scout 
Helicopter  selection.  All  of  them  deal,  and  manage  well, 
with  mix  of  "old"  and  "new”  alternatives. 

Scoping  the  decision  might  be  a  problem  when  the  MAUT 
procedure  is  expected  to  serve  some  levels  of  decision-makers, 
whose  scope  of  decisions  are  not  the  same.  In  such  a  case, 
the  procedure  should  refer  initially  to  the  broadest  scope, 
which  is  usually  determined  by  the  highest  decision-maker. 

The  scope  is  expressed  by  the  range  of  value-dimensions  in¬ 
cluded  in  the  model.  The  construct  of  the  value-dimensions, 
namely,  the  value- tree  should  be  modular,  so  complete  branches 
would  be  eligible  for  elimination  as  a  unit.  Lower  decision 
levels  would  be  presented  only  by  these  models '  branches 
which  are  relevant  to  their  scope  of  decision.  In  most 
cases  this  is  not  difficult  to  do,  since  usually  the  struc¬ 
ture  of  the  objectives  correlates  the  hierarchy  of  the 
decision-makers . 
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The  generation  of  alternatives  is  usually  a  straight¬ 
forward  task  in  the  acquisition  decision.  Cases  of  variation 
on  the  same  basic  alternative,  which  differ  by  various  com¬ 
binations  of  some  components  (for  example,  those  resulting 
from  trade-offs  between  performance  cost  and  schedule) , 
should  be  referred  to  as  separate  contending  alternatives 
[Edwards,  1981]  .  Such  trading-off  processes  can  be  continu¬ 
ous  with  an  infinite  number  of  alternatives;  but  some  discrete 
alternatives  selected  from  the  reasonable  range  might  be  good 
enough  to  represent  the  real  life  selection  options. 

2 .  Step  2 

As  recalled,  at  this  step  stakeholders  are  identified. 

In  the  acquisition  context,  besides  the  DOD  visible  stake¬ 
holders,  they  may  also  be  found  in  Congress,  at  the  states' 
level,  or  in  industry.  Naturally,  in  a  real  case  there  can 
be  too  many  stakeholders  to  permit  a  reasonably  "managed" 
evaluation  model.  So  the  list  must  be  refined,  organized  in 
a  more  or  less  homogeneous  groups  of  individuals  and  organi¬ 
zations,  and  representatives  for  the  actual  interaction  should 
be  identified. 

As  mentioned  before,  the  decision-maker  is  usually  a 
stakeholder  himself  and  as  such,  may  be  addressed  at  this 
phase  like  any  other  stakeholder.  Although  he  has  the  authority 
to  dominate  other  stakeholders,  he  is  wiser  not  to  exert  this 
authority  at  this  early  stage,  because  he  may  lose  inputs 
generated  by  others — inputs  which  may  be  of  importance  to  his 
own  later-on  decision.  Authority  to  dominate  the  process  is 
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suggested  not  to  be  activated  until  the  final  decision 
phases.  On  the  other  hand,  any  decision-maker's  directive 
which  is  a  condition  of  his  acceptance  of  the  methodology  in 
the  first  place,  should  be  implemented  immediately. 

3 .  Step  3 

As  recalled,  at  this  step  elicitation  of  relevant 
value-dimensions  from  the  various  stakeholders  takes  place. 
Usually  the  hierarchy  of  the  value-dimensions  correlates 
with  the  natural  hierarchy  of  the  stakeholders.  It  is  not  so 
difficult  to  generate  a  global  list  of  value-dimensions.  There 
might  be  some  difficulty  in  eliciting  subjective  "political" 
value-dimensions;  on  one  hand,  as  suggested  by  Edwards  (1981) , 
a  stakeholder  should  be  told  that  "it  is  OK  to  be  selfish." 

But  it  is  questionable  whether  a  Congressman  would  openly 
expose  himself  by  articulating  his  "selfish"  interest  to 
provide  his  constituency  a  multimillion  dollar  contract. 

Edwards  (1981)  suggests  the  use  of  proxy  attributes  in  such 
cases.  For  example,  the  Congressman's  objective  to  select 
the  systems  alternative  that  means  contract  awarding  to  his 
constituency  can  be  considered  under  the  "legitimate"  lable. 
"Inter-regional  balance  of  labor  force"  or,  "inter-regional 
balance  of  income  allocation",  or  even  more  indirectly — 
"Prospects  for  Congressional  Approval." 

A  harder  task  is  to  organize  the  value-dimensions  into 
a  value-tree.  This  is  mainly  the  analysis  team's  job.  They 
have  to  make  sure  that  all  important  value-dimensions  are  in 
there,  and  "including  all  relevant  decision  factors  is  a  tough 
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part"  [Appendix  A,  p.  200].  An  effort  should  be  made  to  make 
the  value-tree  as  concise  as  possible.  Some  possible  ways 
to  do  so  have  been  already  mentioned  (see  Chapter  IV.  p.  80) . 

In  cases  where  the  tendency  is  to  reach  objectively  measurable 
attributes  because  the  subjective  scale  seems  to  be  inade¬ 
quate,  attributes  in  addition  to  those  elicited  from  stake¬ 
holders  are  required.  This  attributes'  generation  is  done  by 
experts.  But  to  avoid  too  elaborate  a  value- tree,  a  compro¬ 
mise  is  suggested  as  an  option  by  Keeney  and  Raiffa  (1976, 
p.  45] .  They  say  that, 

...the  vertical  depth  of  the  proliferation  of  the 
hierarchy  does  not  necessarily  force  us  to  quantify 
our  preferences  down  to  this  level  of  detail .  The 
hierarchy  after  a  given  level  may  serve  merely  as 
a  qualitative  checklist  for  items  to  consider. 

But  even  when  there  is  an  effort  to  limit  the  level  of  detail 
of  the  value- tree,  it  still  turns  out  to  be  fairly  elaborate. 
Figures  V-l — V-9  exhibit  an  example  value-tree  for  a  selection 
between  several  hypothetical  advanced  fighter  aircraft  types, 
in  order  to  proceed  with  one  (or  two)  into  Phase  II  (Demon¬ 
stration,  Validation)  of  the  acquisition  life  cycle.  It 
should  be  emphasized  that  this  value-tree  has  not  been  gener¬ 
ated  through  the  procedure  recommended  here  (which  requires  a 
lot  of  manhours  of  a  professional  t-»  It  has  been  constructed 

by  the  thesis  author  just  as  an  illustration  tool.^  Thus 


^In  constructing  this  value-tree  the  author  was  assisted 
by  two  sources: 

(1)  The  value-tree  used  by  Donnell  and  Ulvila  [Decisions 
and  Designs,  1980,  pp.  21-34]  in  their  Decision  Analysis  of 
Advanced  Scout  Helicopter  Candidates. 

(2)  The  questionnaire  input  [Appendix  A,  pp.  237-239]. 


98 


Figure  V-l .  Top-Levels  Structure  of  AFA  Evaluation  Model 
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eptability 


Figure  V-2.  Subdivisions  of  Operational  Acceptability 


Military 


Figure  V-3.  Subdivisions  of  Operational  Effectivness 


Figure  V-4.  Subdivisions  of  RSI  and  Force  Structure  &  Personnel  Impacts 


Technical  System 


Figure  V-5.  Subdivisions  of  Technical  System 
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Figure  V-6.  Subdivisions  of  Attainability 


Life  Cycle  Cost 


Figure  V-7.  Subdivisions  of  Life  Cycle  Cost 


the  details  of  its  construct  are  not  subject  to  discussion 
or  critique  (which  would  be  difficult  anyway,  since  there  are 
no  real  specified  alternatives  behind  this  value-tree  which 
can  be  referred  to) .  As  one  can  notice,  the  tree  is  quite 
elaborate.  (As  a  matter  of  fact,  Donnell  and  Ulvila's  [Deci¬ 
sions  and  Designs,  1980]  Scout  Helicopter  model  goes  even 
further  in  its  attribute  division.)  The  complex  structure 
of  the  illustration  value-tree  stands  in  conflict  with  the 
principle  of  simplicity  that  is  emphasized  throughout  this 
thesis.  Two  basic  approaches  to  cope  with  inevitable  model's 
complexity  can  be  recognized: 

(1)  To  refer  to  only  the  upper  levels  as  the  formal  tree, 
while  all  other  levels  serve  as  a  checklist  for  judgmental 
location  measures. 

(2)  To  retain  the  existing  construct,  but  adjust  the  model's 
complexity  to  the  level  of  decision-maker  whose  decision  the 
model  serves . 

Between  the  two,  an  intermediate  approach  is  selected. 
With  respect  to  the  first  approach,  the  object  of  the  decision 
requires,  and  permits,  subdivisions  down  even  to  6  ,  r  or 

8t^1  levels.  That  is  required  in  order  to  reach  an  objectively 
measured  data  base  for  some  of  the  utility  functions.  This 
is  considered  unreasonable.  Thus,  in  most  attributes,  and 
even  at  the  lowest  subdivisions,  judgment  should  be  applied 
in  partly-subjective  or  purely-subjective  evaluations.  Thus, 
the  question  is  what  should  be  the  optimal  level  at  which  to 
stop  the  attributes  division.  The  complex  acquisition 
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evaluations  require  at  least  4-6  attribute  levels.  That  can 
be  acceptable  if  the  second  approach,  namely,  adjusting  the 
model's  complexity  to  the  decision-maker  level,  takes  place 
as  well.  That  will  be  addressed  in  more  detail  later  in  this 
chapter . 

Another  feature  of  the  value-tree  is  its  modular 
structure.  For  example  it  separates  the  political  category 
(see  1.4  in  Figure  V-l)  from  the  others,  and  enables  the  rank¬ 
ing  order  of  alternatives  to  be  based  on  military  related 
grounds  only.  The  political,  economic,  and  social  analysis 
can  be  done  separately,  and  be  added  to  the  rest  with  suitable 
weights  for  the  final  conclusion. 

As  one  can  notice,  cost  is  evaluated  through  the  four 
main  subcategories  of  the  life-cycle-cost,  which  are  at  the 
entry-level  of  the  cost  category  (see  Figure  V-7) .  This 
stands  in  conflict  to  many  prevailing  analysis  approaches 
which  discount  costs  to  present-value  and  then  refer  to  them 
as  a  unit  dimension.  The  approach  presented  here  offers  a 
more  realistic  handling  of  cost,  since  in  real  life,  cost's 
present-value  minimization  is  not  the  only,  or  major  considera¬ 
tion.  ("As  if  the  only  thing  one  is  worried  about  is  the 
rate  of  interest"  [Edwards,  1981].)  In  many  real  life  cases 
one  may  be  willing  to  accept  smaller  initial  costs,  even  if 
they  mean  eventually  greater  costs  through  time.  Thus  the 
way  cost  is  handled  in  the  example  provides  a  less  mechanical 
approach,  and  greater  emphasis  of  the  cost  over  time  factor. 
Later  on,  at  step  5,  the  cost  will  be  assigned  a  negative 
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utility  which  is  subtracted  from  the  aggregate  utility.  That 
would  cause  the  model  as  a  whole  to  be,  in  a  sense,  a  cost- 
effectiveness  model,  that  measure  the  difference  between  the 
negative  utility  of  cost,  and  the  positive  utility  of  other 
factors  (effectiveness  here  is  synonymous  to  global  utility) . 

Another  issue  that  stands  out  in  the  example  is  the 
existence  of  specific  value-dimensions  for  risks  in  various 
areas  (see  for  example  Figure  V-5,  Technical  Risks,  1.1. 4. 6). 
Those  appear  in  addition  to  the  expected-value  calculation 
(or  estimation)  that  should  take  place  in  any  case  for  the 
uncertain  location  measures.  The  approach  of  having  specific 
attributes  for  risks  is  also  selected  by  Donnell  and  Ulvila 
[Decisions  and  Designs,  1980,  pp.  25-, 32]  in  their  Advanced 
Scout  Helicopter  model.  Edwards  (1981J  refers  to  this  approach 
as  "conceptually  inelegant,"  though  acceptable.  As  a  matter 
of  fact,  it  suits  very  well  the  basic  logic,  suggesting  that 
once  you've  calculated  an  expected-value,  you  deal  with  a 
single  specific  location-measure  for  each  alternative  at  each 
attribute.  But  there  can  be,  for  example,  two  alternatives 
of  which  one  is  an  existing  system  with  well  known  parameters, 
and  the  other  still  on  the  drawing  board  with  highly  variable 
possible  parameters.  Both  alternatives  may  have  equal  measure 
location  over  a  specific  attribute,  and  thus  considered  utility- 
equal  with  respect  to  that  attribute.  But  in  fact,  they  are 
not  perceived  equivalent  since  one  involves  high  risks  and 
the  other  does  not.  A  prudent  decision-maker  would  like  to 
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take  into  account  the  possibility  that  the  new  system  would 
not  perform  as  well  as  the  expected  value  suggests.  The  risk 
dimension  allows  an  explicit  consideration  of  the  uncertain¬ 
ties  involved  with  the  various  alternatives  in  the  value-tree 
Another  way  to  cope  with  the  above  problem  is  through  sensi¬ 
tivity  analysis,  while  mainly  the  variable  and  risky  attri¬ 
butes  are  checked  for  their  impact  on  the  ultimate  outcome 
if  location  measures  are  different  from  the  expected  value. 
Without  overlooking  the  need  for  sensitivity  analysis,  it 
seems  that  the  special  attributes  for  risks  spell  out  the 
risks  involved  to  the  decision-maker  more  clearly.  It  meets 
the  requirement  raised  by  one  of  the  questionnaire  respon¬ 
dents  who  stated: 

A  range  of  subjective  values  and  numerical  estimates 
needs  to  be  presented  to  top  decision-makers,  with 
uncertainties  clearly  spelled  out.  [Appendix  A,  p.  224] 

4 .  Step  4 

As  recalled,  at  step  4  important  weights  are  assigned 
to  the  value-dimensions  at  the  various  levies  of  the  tree. 
Weights,  as  well  as  the  value-dimensions,  are  elicited  from 
the  stakeholders.  As  mentioned  before,  stakeholders  are 
usually  arranged  in  some  sort  of  a  hierarchy  that  correlates 
the  hierarchy  of  the  value-dimensions.  This  feature  suggests 
that  specific  weights  are  elicited  from  specific  stakeholders 
Not  all  stakeholders  are  supposed  to  refer  to  all  weights  of 
all  value-dimensions.  The  use  of  the  value-tree  construct 
for  the  weighting  procedure  (see  Chapter  IV,  p.  83). 


provides  a  convenient  framework  for  weighting  by  small  groups 
of  weights  distinguished  by  their  horizontal  belonging  to 
specific  attributes  levels,  and  by  their  vertical  relation¬ 
ships  to  specific  branches.  Like  in  the  attributes  elicita¬ 
tion,  the  weights  for  many  of  the  entry-level  attributes  in 
the  acquisition  case  should  be  done  preferably  by  experts 
rather  than  by  stakeholders. 

The  questionnaire  respondents  give  priority  to  the 
decision-maker  in  weights  determination.  It  seems  that  they 
are  not  aware  of  the  possibility  of  elicitations  from  stake¬ 
holders  (and  the  questionnaire  itself  did  not  provide  it) . 

Even  with  a  broader  weighting  source,  the  decision-maker  should 
retain  the  last  word  in  resolving  unsettled  disputes  over 
the  value-tree  construction  and  weighting. 

The  notion  of  disagreements  over  weights  brings  about 
the  issue  of  using  several  weight  sets.  As  revealed  by  the 
questionnaire  [Appendix  A,  pp.  231-236] ,  there  are  different 
sets  of  relative  weights  for  different  decision-makers  levels. 
Those  differences  are  very  natural.  It  is  quite  obvious  that 
SecDef,  for  example,  would  assign  higher  weights  to  political 
value-dimensions  than  the  Service.  While  the  modular  struc¬ 
ture  of  the  value-tree  may  provide  resolution  for  different 
scopes  at  the  various  levels,  it  does  not  provide  a  resolution 
for  differences  in  weighting  of  the  same  spectrum  of  value- 
dimension.  In  fact,  the  same  factor  areas  considered  at 
DSARC  or  SecDef  level  are  considered  also  at  the  top  Service 
Department  levels,  though  not  with  identical  importance  weights 


For  example,  the  questionnaire  indicates  that  political  and 
socio-economic  aspects  are  also  considered  at  the  Service 
level,  although  by  different  importance  weights  compared  to 
OSD  top- levels.  As  mentioned,  these  differences  cannot  be 
resolved  by  the  modular  structure.  Another  approach  should 
be  selected:  First,  weights  should  be  reduced  to  very  few 
sets.  Agreements  within  the  main  groups  involved  such  as  OSD, 
Service,  or  Industry  may  be  reached.  After  all,  as  said  by 
Edwards  (1977,  p.  248),  "Every  boss  has  a  boss,  and  every 
one  attempts  to  take  the  values  of  his  superior  into  account 
in  his  own  decisions."  And  really,  the  questionnaire  reflects 
only  reasonably  moderate  deviation  between  the  weight  sets  of 
the  four  DOD  top-level  acquisition  decision-makers  (see  Appen¬ 
dix  A,  p.  233)  .  Then,  as  previously  mentioned,  there  is  an 
option  to  present  separate  evaluation  outcomes  as  developed 
by  using  several  major  weight  sets,  while  each  represents  a 
major  group  of  stakeholders.  The  judgment  between  them  is 
left  for  the  ultimate  decision-maker. 

5 .  Step  5 

As  recalled,  at  this  step  utility  functions  are  con- 
structured,  and  each  alternative  is  evaluated  as  to  how  well 
it  performs  at  each  of  the  value-dimensions.  The  construction 
of  utility  functions  is  a  complex  process  and  each  of  them 
should  get  individual  treatment,  using  a  suitable  approach. 
Naturally,  the  utility  functions  for  objectively  measured 
technical  attributes  should  be  constructed  by  different 
methods,  consulting  different  experts  than  the  purely  subjec¬ 
tive  ones.  There  is  no  intention  to  describe  in  detail  these 


methods,  whose  use  should  be  considered  on  a  case  by  case 
basis.  In  general,  there  must  be  a  striving  for  linear  func¬ 
tions,  as  suggested  by  Edwards  and  many  others  (see  Chapter  IV. 
p.  84) .  For  the  purely  subjective  attributes  an  even 
simpler  way  may  be  appropriate.  Instead  of  a  continuous 
linear  function,  some  5  or  10  discrete  points  on  it  can  be 
selected,  stated  in  terms  of  "excellent,"  "very  good,"  etc., 
or  by  a  broader  verbal  formulation  of  quality  grades  (for 
example,  see  Larichev,  (1977,  p.  199)).  Each  such  verbal 
utility  grade  has  an  associated  numerical  value  over  the  utility 
function  scale,  so  it  enters  the  model  mathematical  procedure. 

In  constructing  the  utility  functions,  the  notion  of 
thresholds  arises.  Several  of  the  questionnaire  respondents 
require  the  procedure  to  maintain  a  provision  for  handling 
thresholds.  If  the  relative  scaling  method  recommended  by 
Donnell  and  Ulvila  is  selected  (see  Chapter  IV,  pp.  84-86) ,  no 
inherent  threshold  handling  capability  is  provided.  Accord¬ 
ing  to  Donnell  (1981)  the  solution  for  that  problem  is  to  test 
the  alternatives  for  meeting  thresholds  external  to  the  model. 
Only  those  who  pass  the  test  enter  the  formal  evaluation. 

But  since  a  threshold  is  not  usually  an  absolutely  unchange¬ 
able  entity,  interesting  alternatives  may  get  lost.  For 
example,  Keeney  and  Nair  (1977,  p.  299)  decided,  in  their 
Nuclear  Power  Plant  Sites  analysis,  not  to  exclude  areas  merely 
because  they  failed  just  under  or  over  cutoff  level  on  one 
criterion.  Using  the  threshold  as  a  "true  zero"  point  in  the 


absolute  or  the  ratio  scaling  methods,  enables  implementing 
what  Edwards  and  Newman  call  "outside-the-range  objects  scor¬ 
ing,"  i.e.,  scores  that  below  zero  are  legal,  "so  long  as  the 
meaning  of  the  ranges  is  kept  clearly  in  mind"  [Edwards  and 
Newman,  1980,  Chapter  5,  p.  51.  Thus,  the  strong  requirement 
for  thresholds  in  real  life  cases  weakens  somewhat  Donnell's 
strong  support  for  relative  scaling.  But  as  indicated  by 
Donnell  (1981)  himself,  different  scaling  methods  can  be  used 
in  the  same  model,  as  long  as  a  complete  branch  (or  category) 
uses  the  same  scaling  approach.  Thus,  for  threshold-sensitive 
value-dimensions  such  as  cost,  the  absolute  scaling  is  pre¬ 
ferred,  while  in  other  cases  the  relative,  or  ratio,  scales 
are  good  enough. 

Another  issue  associated  with  step  5  is  the  question 
of  who  should  evaluate  the  alternatives  and  determine  their 
individual  scores  over  the  various  utility  functions.  Ac¬ 
cording  to  Edwards  (1981)  that  is  an  experts’  job.  Truely, 
looking  at  the  nature  of  the  entry-level  attributes,  almost 
none  of  them  are  separately  a  subject  for  high-level  judgment. 
Furthermore,  Edwards  warns  against  letting  stakeholders  do  the 
scoring.  According  to  Edwards,  models  are  more  sensitive  to 
scores  than  to  weights,  and  if  there  is  a  place  for  biasing  a 
model,  scoring  is  that  place.  The  questionnaire  respondents 
reveal  clear  reservations  towards  experts'  evaluations  [Appen¬ 
dix  A,  p.  219).  (The  Israelis  are  much  more  open  to  experts' 
aid.)  In  general,  the  respondents  would  like  to  see  extensive 
involvement  of  the  decision-maker  in  the  scoring  process,  but 
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this  is  not  very  practical.  In  any  case,  the  message  is 
clear:  they  don't  want  the  evaluations  of  some  medium  or 
low-level  experts  (those  can  be  operational  people  as  well 
as  technical  ones)  to  be  the  only  input  to  the  procedure.  It 
is  not  acceptable  that  once  the  low  level  evaluations  take 
place,  they  are  pluged  into  the  model  and  automatically  run 
to  the  final  outcome.  The  feeling  is  that  some  sort  of  higher 
level  reviews  of  the  model  are  required  before  it  is  presented 
to  the  ultimate  deci.*- wn-maker .  Two  ways  are  suggested  to 
meet  this  requj.  c :  . 

(1)  Limit  the*  v-%.>.i.-*-tree  to  some  relatively  high-level  of 
value-dimension.  Then  the  evaluation,  naturally  more  sub¬ 
jective,  will  be  done  by  higher  level  evaluators. 

(2)  Review  the  evaluations  done  by  "experts"  by  higher- 
level  "judges."  Each  will  take  care  of  a  specific  branch  that 
is  in  his  area  of  expertise.  Those  higher-level  "judges"  will 
be  eligible  to  override  evaluations  that  they  do  not  accept. 
The  question,  of  course,  is  who  are  these  "judges?"  It  is 
hard  to  imagine  such  an  evaluation  process  accomplished  com¬ 
pletely  independently  of  the  stakeholders,  by  some  'bbjective 
evaluators."  No  doubt,  the  Service  should  be  the  major  evalua¬ 
tor  for  any  military  related  aspects.  To  counter  possibility 
of  biased  evaluations  there  are,  after  all,  some  "insurances" 
while  the  MAUT  procedure  is  used.  The  procedure  cannot  be 
completely  bias-proof,  but  it  reduces  such  possibilities,  by 
its  following  features: 


(1)  The  very  fact  that  "subdecisions"  are  done  separately 
for  each  value-tree  module  (or  branch)  reduces  the  possibility 
of  overall  bias. 

(2)  The  procedure  forces  people  to  articulate  their  impor¬ 
tance  weights,  preferences,  evaluations,  and  hence  automati¬ 
cally  increases  transparency  of  the  alternatives  presented 
[Edwards,  1981] .  This  notion  is  supported  by  the  Navy  Program 
Manager  Guide  [Naval  Material  Command,  1980,  pp.  3-25],  which 
views  the  achievement  of  a  much  greater  degree  of  objectivity 
in  the  evaluation  process  than  might  otherwise  be  expected,  as 
one  of  MAUT's  greatest  advantages: 

When  opinions  are  displayed  and  critiqued,  narrow 
minded  orientations  give  way  to  more  balanced  outlook. 

This  is  true  at  any  level  of  decision-making.  While  the  sub¬ 
decision-makers  or  the  "experts"  for  specific  categories  are 
in  most  cases  inevitably  associated  with  one  or  another  party 
of  stakeholders,  it  is  important  that  the  analysis  team,  which 
runs  the  procedure,  be  independent  or  report  directly  to  the 
top  decision-maker  office.  That  can  be  another  safeguard  for 
objectivity.  All  this  is  unnecessary  in  an  environment  of 
greater  mutual  credibility  than  the  one  described  by  the  ques¬ 
tionnaire  comments.  In  environments  in  which  the  danger  of 
biased  evaluations  is  not  sc  significant,  the  program  office 
is  the  natural  home  for  running  the  MAUT  procedure. 

6 .  Step  6 

As  recalled,  at  step  6  the  weighed  location  measures 
are  aggregated  into  a  single  figure  of  merit,  i.e.,  an  overall 
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utility  figure  for  each  alternative.  The  one  with  the  highest 
figure  is  the  preferred  alternative  (according  to  the  MAUT 
procedure  only,  of  course) .  The  aggregation  rule  recommended 
here  is  the  additive  rule,  as  preferred  by  most  MAUT  practi¬ 
tioners  (see  Chapter  IV,  p.  76) .  The  aggregation,  by  itself, 
is  a  relatively  simple  calculation,  but  it  is  not  yet  a  deci¬ 
sion.  It  is  the  decision-making  process  that  will  be  addressed, 
rather  than  the  mathematical  computation  of  the  aggregate 
utility.  Some  highlights  of  the  decision-making  process  asso¬ 
ciated  with  the  MAUT  procedure  implementation  are  presented 
here . 

As  mentioned  in  the  previous  section,  various  branches 
of  the  model  may  be  reviewed  by  separate  sub-decision-makers 
that  are  authorized  to  do  so.  But  when  the  decision  reaches 
top- levels  such  as  (S)SARC,  Service's  Source  Selection  Board, 
(S)Sec,  DSARC  or  SecDef,  the  presentation  should  include  all 
the  model's  major  categories.  However  that  might  be  an  ex¬ 
hausting  presentation,  unless  the  model's  complexity,  or  in 
other  words — level  of  detail,  is  adjusted  to  the  position  and 
the  personal  preferences  of  the  decision-maker  involved  (recall 
Chapter  V,  p.  108)  .  The  actual  meaning  of  this  adjustment  is 
the  model's  presentation  that  starts  from  some  selected  level 
of  attribute  subdivision  and  up.  The  lowest  level  presented 
should  not  necessarily  be  uniform  at  all  branches.  It  depends 
on  how  deep  the  decision-maker  is  willing  to  go  down  at  each 
of  them.  In  some  instances  he  may  be  willing  to  go  down  to 
the  entry-level  in  order  to  examine  the  utility  functions,  and 
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in  another  he  may  be  satisfied  by  reviewing  to  only  the  second 
or  third  level.  The  model's  presentation  should  be  supple¬ 
mented  by  subaggregation  output  for  each  desired  level  of 
attributes  (see  Chapter  IV,  p.  87) .  In  addition,  the  decision¬ 
maker  should  be  verbally  briefed  about  the  underlying  data, 
and  the  rationale  behind  the  model's  construct.  In  such  a  way, 
the  decision-maker  has  a  good  picture  of  the  main  objectives 
or  value-dimensions  of  his  interest  and  their  relative  weights; 
the  relative  location  of  each  alternative  over  the  major  cate¬ 
gories  (by  the  subaggregations) ;  and  a  more  general  idea  about 
the  construct  and  the  utility  functions  at  those  levels  which 
are  not  presented. 

The  notion  of  presenting  the  basic  data,  how  the  evalu¬ 
ation  was  generated,  and  the  MAUT  procedure  construct  rationale, 
as  a  supplement  to  the  model's  numerical  parameters,  is  strongly 
recommended  by  the  questionnaire  respondents  [Appendix  A,  pp.  224, 
226],  Another  questionnaire  input  is  that  subaggregations 
should  not  replace  the  presentation  of  the  single  figure  aggre¬ 
gated  global  utility.  The  latter  is  definitely  required,  as 
reflected  in  the  answers  to  Q28  of  the  questionnaire  [Appendix 
A,  p.  214 ]  .  That  is  different  from  methods  found  in  the  litera¬ 
ture,  which  recommend  listing  the  characteristics  and  impacts 
of  the  various  alternatives,  and  leave  the  task  of  ranking  to 
the  judgment  and  intuition  of  the  decision-maker.  Those  methods, 
such  as  "Goeller  Scorecard,"  sometimes  supplemented  with  "color 
coding,"  have  been  suggested  for  cases  where  "classical”  ap¬ 
proaches  like  cost-benefit  were  not  able  to  prepare  unambiguous 
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ranking  of  alternatives  (Quade,  1979,  p.  59].  But  that  is  not 
the  case  with  MAUT,  which  overcomes  many  of  the  "classical" 

analysis  method’s  weaknesses. 

Some  of  MAUT's  advantages  noted  in  the  literature 
were  also  pointed  out  by  the  questionnaire  respondents.  Ac¬ 
cording  to  the  questionnaire  [Appendix  A,  p.  211-212],  the 
following  benefits  might  result  if  MAUT  is  used  in  the  acqui¬ 
sition  decision  sessions: 

(1)  It  allows  better  communication  and  understanding  among 
the  people  involved. 

(2)  It  has  the  potential  to  focus  discussion  on  the  impor¬ 
tant  issues. 

(3)  It  assists  the  people  to  put  rigor  into  their  line  of 
thinking. 

Of  course,  as  in  any  discussion,  these  points  can  be 
missed  if  the  discussion  is  badly  managed.  As  a  consequence, 
some  of  the  questionnaire  respondents  are  reluctant  to  use 
MAUT  because  of  the  worry  that  the  procedure  may  shift  a  dis¬ 
cussion  to  unnecessary  arguments  on  weights  and  scores  [Appen¬ 
dix  A,  p.  203] •  The  answer  to  this  worry  is  very  simple: 
"don't  mismanage"  [Edwards,  1981]. 

An  interesting  issue  is  the  case  that  the  procedure 
would  result  in  an  outcome  which  the  decision-maker  is  not 
willing  to  accept,  whatever  the  reason — "gut  feeling,"  or 
more  explainable  reason.  However,  some  of  the  questionnaire  s 
respondents  say,  "then  he  may  be  baring  his  belly  to  his  foes" 
(by  agreeing  to  use  the  procedure,  R.G.)  [Appendix  A,  p.  206] . 
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There  are  several  ways  to  avoid  this  problem.  First,  the 
preliminary  acceptance  of  the  procedure  by  the  decision-maker 
reduces  the  probability  of  the  outcome  being  rejected  later 
in  the  process.  If  the  decision-maker  participates  in  the 
model's  construction  and  guides,  or  even  overrules  parameters 
decisions,  that  probability  is  even  smaller.  Thus,  it  is 
essential  for  the  MAUT  procedure  success  to  have  the  decision¬ 
maker's  involvement  from  its  early  phases.  Second,  it  should 
be  recognized  that  it  is  perfectly  legal  for  the  decision¬ 
maker  to  change  the  model's  parameters  according  to  his  own 
perception.  After  all,  he  is  the  one  that  should  make  the 
ultimate  decision,  and  the  responsibility  is  always  on  his 
shoulders  [Trimble,  1981].  The  questionnaire  responses  strongly 
support  the  decision-maker's  authority  to  change  the  model's 
parameters  [Appendix  A,  p.  214] .  That  is  also  acceptable  by 
practitioners  such  as  Edwards  (1981) .  In  the  worst  case,  the 
decision-maker  would  adjust  the  model  "brutally"  to  fit  his 
own  selection.  Even  in  such  a  case,  the  model  usage  has  been 
probably  contributed  to  the  decision-making:  first,  it  served 
as  an  input  that  after  all  cannot  be  ignored.  Second,  in 
changing  the  model's  parameter^  the  decision-maker  is  forced 
to  articulate  his  own  parameters  in  terms  of  weights  and  scores, 
which  puts  rigor  into  the  decision-making.  But  in  real  life, 
the  decision-maker  interaction  with  MAUT  model  is  not  so 
brutal.  A  more  "diplomatic  approach"  is  suggested  by  one 
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Usually,  I  don't  think  the  decision-maker  will 
openly  or  overtly  change  weights  and  scores.  To 
do  so  at  the  "end"  only  invites  allegations  of 
improper  rigging  of  selection.  [Appendix  A,  p.  218] 

Edwards  (1981)  adds  that  the  decision-maker  should  challenge 
the  analysts  if  he  thinks  the  models  outcomes  are  incorrect. 

As  a  consequence,  a  wise  decision-maker  would  not  allow  a 
model  to  provide  a  "wrong"  result  in  the  first  place.  By 
following  its  preparations  he  may  affect  it  in  the  early 
stages.  But  if  a  conflict  arises  with  his  own  attitudes  at 
the  final  stages  after  all,  he  would  probably  send  the  analysts 
back  for  a  "second  guess"  while  pointing  out  what  he  thinks  is 
wrong.  That  should  be  the  real  meaning  of  interaction  between 
the  decision-maker  and  the  MAUT  procedure.  True,  in  some  in¬ 
stances  time  constraints  do  not  allow  the  full  interaction  as 
suggested  here,  but  some  level  of  interaction  must  exist  if  the 
procedure  is  selected  as  the  decision  tool.  The  acquisition 
decisions  are  important  enough  to  justify  several  iterations 
of  the  evaluation  model  presentation  to  the  top  decis ion-makers , 
in  order  to  obtain  their  acceptance  of  the  evaluation,  and 
the  procedure ' s  outcome . 

Finally,  it  has  to  be  emphasized,  at  noted  by  some  of 
the  questinnaire  respondents  [Appendix  A,  p.  202]  and  the 
interviewees  (for  example  Trimble  (1981)  and  Edwards  (1981)) 
that  the  MAUT  procedure  should  never  dominate  the  decision. 

The  final  decision  is  the  exclusive  responsibility  of  the 
decision-maker,  and  no  model's  outcome  should  be  an  excuse 
for  a  bad  decision.  He  should  exploit  wisely  whatever  available 
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from  the  MAUT  evaluation  procedure,  but  he  should  never  let 
it  replace  him.  In  that  sense,  the  MAUT  procedure  is  only  a 
"tool, "  as  defined  by  many  of  the  espondents  [Appendix  A, 
p.  202] •  Truely,  it  does  no  more  than  provide  a  mechanism 
whereby  proposed  alternative  concepts  can  be  evaluated 
[Navy  Material  Command,  1980,  pp.  3-25).  Though  accepting 
this  view  as  noted  above,  the  thesis  analysis  suggests  that 
this  mechanism  should  be  used  as  the  major  framework  for  the 
evaluation  process  and  decision-making. 

7 .  Step  7 

As  recalled,  step  7  involves  a  sensitivity  analysis. 
This  is  an  important  part  of  MAUT's  implementation,  and  its 
results  should  be  an  integral  part  of  the  model  presentation 
to  decision-makers  at  the  various  levels. 

As  mentioned  in  Chapter  IV,  the  most  important  kind 
of  sensitivity  to  look  at  is  sensitivity  to  weights.  Sensi¬ 
tivity  analysis  of  weights  can  replace  or  complement  presenta¬ 
tion  of  several  weight  sets  in  order  to  represent  separately 
the  major  groups  or  stakeholders  involved  in  the  decision. 
Instead  of  several  weight  sets,  only  one  is  used,  serving  as 
a  basis  on  which  changes  of  weightings  are  tested  and  analyzed. 
The  ranges  of  weights  to  be  tested  may  be  reduced  by  applying 
some  "rules  of  thumb,"  which  define  the  relationships  between 
weight  perceptions  of  groups  or  organizations.  Such  rela¬ 
tionships  between  weights  sets  made  at  several  decision  levels 
within  the  DOD  are  presented  and  analyzed  in  Appendix  A 


C.  MAUT  PROCEDURE  AS  AN  ONGOING  PROCESS 

The  basic  ideas  of  the  MAUT  procedure  as  an  ongoing  proc¬ 
ess  have  been,  in  fact,  introduced  by  means  of  the  detailed 
comments  about  its  implementation  in  the  acquisition  decision¬ 
making  process.  This  section  attempts  to  combine  those  ideas 
into  a  complete  concept.  The  idea  is  to  use  MAUT  procedure 
not  only  as  a  "one  shot"  evaluation  tool,  but  as  an  ongoing 
framework  for  evaluation  through  the  whole  acquisition  life 
cycle.  The  value-tree's  various  branches  may  be  used  as  inde¬ 
pendent  evaluation  models  for  various  aspects  of  the  program. 
When  combined  altogether,  they  would  provide  the  over-all 
evalaution.  The  model  as  a  whole  or  its  separate  modules  may 
serve  as  the  major  tool  for  any  current  review  or  decision¬ 
making  session,  and  not  necessarily  for  top-level  decision¬ 
makers.  It  should  be  updated  and  changed  in  both  construct 
and  parameters  as  more  information  accumulated  through  time. 
Consequently,  it  should  be  maintained  currently  by  a  suitable 
team.  (After  procedure  initiation,  its  maintenance  should  not 
be  a  full  time  job  for  the  team  members.)  The  natural  home 
for  such  a  team  might  be  the  program  office.  But  the  team's 
nucleus  should  consist  of  independent  personnel  who  report 
essentially  to  the  top  decision-maker.  That  is  required  as  a 
safeguard  against  biased  evaluations. 

In  such  a  way,  an  up-to-date  evaluation  and  comparison 
between  program  alternatives  can  be  provided  at  any  time,  and 
not  only  towards  Milestone  decisions.  From  the  methodological 
aspect,  MAUT,  as  a  modular  easy-to-update  procedure  fits 
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the  function  of  an  ongoing  evaluation  and  comparison  framework 
very  well. 

D.  CONCLUSIONS 

1 .  The  Chapter's  Conclusions 

The  major  conclusions  of  this  chapter  can  be  summarized 
as  follows: 

(1)  In  any  case  of  using  MAUT  for  major  acquisition  decision¬ 
making,  skilled  practitioner's  consultation  is  required. 

(2)  The  decision-maker's  acceptance  of  MAUT  procedure  as 
the  major  evaluation  and  presentation  tool  should  be  assured 
in  early  phases  of  the  implementation. 

(3)  MAUT  procedure  is  very  versatile  and  fits  various  types 
of  acquisition  decisions  such  as  Milestones  decisions  (mainly 
I  and  II) ,  source  selection,  and  other  status  reviews  and 
sub-decisions  at  the  various  decision-making  levels. 

(4)  MAUT  procedure  is  applicable  to  the  evaluation  and 
selection  between  unlike  systems  (such  as  missile  vs.  air¬ 
craft)  or  between  "old  vs.  new"  systems.  Though  such  compari¬ 
son  raises  some  practical  problems,  they  should  be  and  can 

be  resolved. 

(5)  The  modular  construct  of  MAUT's  value- trees  permits 
implementation  of  separate,  partial  evaluations  for  components 
of  the  overall  decision. 

(6)  Value-dimensions  should  be  elicited  from  all  major 
stakeholders,  including  the  decision-maker  himself.  Value- 
dimensions  should  also  include  "selfish"  political  factors, 
perhaps,  in  the  shape  some  surrogate  (proxy)  attribute. 
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(7)  Inspite  of  simplification  efforts,  value- trees  for 
major  acquisition  decisions  inevitably  turn  out  to  be  quite 
elaborate . 

(8)  Adjustment  of  model  complexity  to  top-level  decision¬ 
maker  levels  can  be  done  by  a  partial  presentation  of  the 
model,  starting  at  some  value-dimension  level  and  going  up  to 
the  prime  objective.  Such  a  presentation  is  supplemented 

by  subaggregations,  underlying  data  and  methodology  rationale 
briefings . 

(9)  There  might  be  differences  in  weighting  between  stake¬ 
holders  or  levels  of  decision-maker .  When  they  cannot  be  re¬ 
solved,  several  aggregations,  using  several  weights  sets,  can 
be  presented,  being  subject  to  the  decision-maker  judgment. 

(10)  To  avoid  leaving  the  scoring  of  the  alternatives  only 
to  "experts,"  various  branches  of  the  model  should  be  re¬ 
viewed  by  sub-decision-makers,  who  would  be  authorized  to 
exert  their  judgment  in  overruling  the  experts'  evaluations. 

(11)  The  scaling  method  should  meet  the  requirement  for 
threshold  inclusion  for  some  of  the  attributes. 

(12)  While  "experts,"  evaluators  and  sub-decision-makers 
are  inevitably  identified  with  some  stakeholders'  organiza¬ 
tions  (e.g.,  a  Service),  the  analysis  team  should  report  tc 
the  final  decision-maker,  as  a  safeguard  for  unbiased 
evaluations . 

(13)  The  decision-maker  is  eligible  to  change  the  model's 
construct  and  parameters  according  to  his  own  subjective 
judgment.  A  "wise"  decision-maker  should  interact  with  the 
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model  during  its  development  to  assure  that  conflicts  would 
not  be  exposed  only  at  the  final  stages  of  the  decision 
process.  If  there  is  a  conflict  between  his  preference  and 
the  model's  outcome,  he  had  better  challenge  the  analysts  to 
find  out  what  was  wrong,  rather  than  "brutally"  change  the 
model,  or  worse,  just  ignore  it. 

(14)  The  MAUT  procedure  is,  after  all,  a  decision  "tool," 
although  as  suggested  here — a  central  one,  and  should  not 
replace  the  decision-maker  who  always  has  the  ultimate  responsi¬ 
bility  for  the  decision. 

(15)  MAUT  procedure  is  suggested  to  serve  as  an  ongoing 
comprehensive  evaluation  base  through  the  whole  acquisition 
life  cycle. 

2 .  A  Check  for  Meeting  the  Required  Criteria 

At  this  stage,  after  having  introduced  MAUT's  charac¬ 
teristics  in  general,  and  its  implementation  issues  for  acqui¬ 
sition  decisions  in  particular,  it  is  appropriate  to  go  back 
and  check  how  well  the  sugtested  methodology  meets  the  criteria 
set  at  the  conclusion  of  Chapter  II  (see  pp.  45-47)  .  This  check 
provides  the  following  observations:  (Comment:  Order  of 
items  here  correlates  the  order  of  the  requirements  in  Chapter 
II.) 

(1)  MAUT  takes  into  account  the  informal  decision  process 
by  identifying  the  informal  stakeholders  and  by  eliciting 
from  them  value-dimensions  and  weights. 

(2)  MAUT  by  its  very  nature  is  a  multi-objective  decision 
technique. 
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(3)  The  value-tree  construct  allows  sorting  of  the  fac¬ 
tors,  focusing  on  the  important  ones,  and  eliminating  the 
irrelevant  ones.  The  partial  presentation  described  in  this 
chapter  enalbes  any  desired  level  of  detail  of  presentation 
for  decision. 

(4)  The  concise  but  comprehensive  alternatives'  presenta¬ 
tion  through  the  MAUT  procedure  makes  the  decision-maker  more 
capable  to  evaluate  and  select  alternatives. 

(5)  The  rigor  and  discipline  imposed  by  MAUT  procedure 
inherently  increases  model  transparency  and  by  that  counter 
biased  presentation  tendencies. 

(6)  The  suggestion  that  MAUT  be  an  ongoing  evaluation 
framework  provides  the  element  of  evaluation  updating. 

(7)  MAUT  can  serve  as  a  comprehensive  evaluation  tool 
including  political,  economic  and  social  issues. 

(8)  MAUT  is  a  quantitative  technique  which  is  capable  of 
handling  subjective,  judgmental  consideration. 

(9)  The  "wise"  interaction  of  the  decision-maker  with  the 
model  may  reduce  the  possibility  of  conflict  between  his  deci¬ 
sion  and  the  model  outcome,  and  thus  reduce  the  possibility  of 
using  the  model  against  his  decision. 

(10)  The  subjective  evaluations  involved  in  MAUT  include 
also  “rules  of  thumb"  and  descriptive  elements,  but  putting 
them  into  order  causes  a  better  exploitation  of  the  decision¬ 
maker's  skills. 

(11)  Although  top-level  decision-makers  remain  highly  depen¬ 
dent  on  their  subordinates  evaluation,  the  comprehensive 
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decision  presentation  allows  them  better  evaluations  of  their 
own. 

(12)  The  modular  construct  of  MAUT  model  enables  working 
through  the  organizational  functional  and  hierarchical  structure. 

(13)  The  MAUT  procedure  provides  fair  and  ordered  repre¬ 
sentation  of  political  powers  and  their  attitudes. 

(14)  The  framework  of  the  procedure's  presentation  compen¬ 
sates,  to  a  certain  degree,  personal  differences  of  charisma, 
presentation  skills  and  alike. 

(15)  The  modular  construct  of  a  MAUT  model  enables  adjust¬ 
ment  of  the  presentation  complexity  to  the  individual 
decision-maker . 

(16)  The  implementation  recommendations  of  this  chapter  meet 
the  requirement  of  the  model's  early  acceptance  by  the  decision¬ 
maker,  and  the  latter's  interaction  with  the  model  through 

its  various  phases. 
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VI .  CONCLUSIONS  AND  RECOMMENDATIONS 

A.  CONCLUSIONS 

The  major  concern  driving  this  research  has  been  the 
desire  to  improve  top-level  acquisition  decision-making.  The 
following  major  problems,  deficiencies,  and  difficulties  of 
the  present  acquisition  decision-making  situation,  have  been 
identified: 

(1)  Difficulties  in  circumscribing  the  decision  factors 
and  consolidating  them  all  for  a  final  decision.  This  stems 
from  the  complex,  multi-objective  nature  of  the  acquisition 
decisions . 

(2)  Lack  of  balance  of  political  impacts  of  individuals, 
groups  and  organizations,  imposed  upon  the  decision-making 
process.  Disorder  in  the  representation  of  those  who  have 
stakes  in  the  decision. 

(3)  Existence  of  phenomena  such  as  biased  decision  presen¬ 
tation,  which  is  amplified  by  the  inevitable  reliance  of  top- 
level  decision-makers  on  their  subordinates '  evaluations . 

(4)  Aversion  to  the  usage  of  quantitative  techniques  as 
top-level  decision  aids  which  stems  mainly  from  the  lack  of 
confidence  in  the  capability  of  these  techniques  to  incorporate 
intuitive,  judgmental  issues. 

In  the  search  for  decision  techniques  that  may  improve  the 
above  deficiencies,  the  thesis  concludes  that  there  are  in 
existence  decision  techniques  which  are  capable  of  incorporating 
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subjective,  judgmental  issues,  and  of  combining  them  with 
objectively  measured  data.  After  analyzing  the  nature  of  the 
acquisition  decision,  a  conclusion  is  drawn  that  the  above- 
mentioned  techniques  might  be  of  great  help  to  these  decisions 
Among  the  decision  techniques  MAUT  stands  out  as  the  most 
preferred  by  theoreticians,  as  well  as  actual  users. 

After  a  review  of  the  basic  concepts  of  MAUT,  recommenda¬ 
tions  for  implementation  of  the  procedure  are  suggested. 

The  major  conclusions  for  implementation  are: 

(1)  MAUT,  as  a  versatile  method,  is  compatible  with  a 
broad  spectrum  of  acquisition  decision  types. 

(2)  MAUT's  model  for  major  system  acquisition  decisions 
is  inevitably  complex,  but  its  modular  construct  allows 
"vertical"  and  "horizontal"  cuts,  in  order  to  adjust  the 
model's  scope  and  level  of  detail  to  the  decision-maker's 
position  and  personal  preferences.  Ways  to  do  it  are 
suggested. 

(3)  A  "fair"  representation  of  stakeholders  can  be  achieved 
by  their  substantial  involvement  during  the  model ' s  construct 
process  (value-dimensions  and  weights  elicitations)  and 
during  the  model's  presentation  (by  presenting  several  aggre¬ 
gations  with  several  weights  sets) . 

(4)  The  ultimate  decision-maker  should  "interact"  with  MAUT 
procedure  from  the  beginning  by  preliminary  acceptance, 
participating  in  the  value-dimensions  and  weights  elicitations 
guiding  the  model's  construction  and  interacting  with  the 
model  in  the  presentation  for  the  decision. 
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As  explained  at  the  end  of  Chapter  V,  the  MAUT  procedure 
satisfies  to  a  great  extent  the  criteria  established  in 
Chapter  II,  and  thus,  offers  solutions  to  many  of  the  defi¬ 
ciencies  and  difficulties  of  the  present  decision  process. 
However,  MAUT  is  by  no  means  a  “magic  prescription"  for  all 
the  existing  decision  process'  weaknesses:  First,  there  are 
still  questions  of  how  to  generate  reliable  data  for  its 
various  inputs.  Second,  its  success  is  sensitive  to  the 
quality  of  the  implementation  process  in  each  individual 
case.  Third,  although  capable  of  incorporating  subjective 
judgment,  it  cannot  generate  it.  It  also  cannot  express  and 
represent  all  the  delicate  components  of  human  thought.  The 
above  mentioned  issues  imply  an  important  conclusion:  MAUT 
is, after  all,  a  "tool,"  and  should  never  dominate  the  decision. 
It  is  the  decision-maker  who  has  the  responsibility  and  the 
authority  to  make  the  decision,  and  he  is  accountable  for  it. 

The  message  of  this  thesis  is  that  usage  of  MAUT  proce¬ 
dures  as  the  major  evaluation  framework  and  the  center  of  the 
decision  presentation  for  the  highest  decision  levels,  would 
improve  the  efficiency  of  the  process,  increase  objectivity, 
contribute  to  top-levels  decision-making  capability,  and 
consequently  bring  about  better  decisions. 

B .  RECOMMENDATIONS 

(1)  The  MAUT  procedure  should  be  used  as  the  major  tool 
for  presenting  and  evaluating  alternatives  for  acquisition 
decisions  at  the  top  levels.  The  procedure  should  include 
political,  economic  and  social  factors  in  addition  to  the 


132 


directly  relevant  factors  such  as  military  worth  or  cost. 

It  should  provide  fair  representation  of  the  major  stakeholders 
involved  in  the  decision. 

(2)  For  each  major  system  MAUT  should  serve  as  comprehen¬ 
sive,  ongoing,  updated  evaluation  framework  through  the  whole 
acquisition  life  cycle.  On  the  base  of  this  evaluation  frame¬ 
work,  major  decisions  as  well  as  sub-decisions  would  be  made. 

This  ongoing  evaluation  framework  should  be  constructed,  main¬ 
tained  and  reviewed  by  staff  members  and  sub-decision-makers 
through  the  functional  and  hierarchical  organizational  struc¬ 
ture.  At  the  nucleus  of  the  evaluation  framework  would  be 
an  analysis  team  that  may  be  located  at  the  program  office, 
but  would  be  formally  subordinate  to  the  ultimate  decision-maker. 

For  more  detailed  description  of  implementation  issues, 
reference  may  be  made  to  the  conclusions  as  presented  in 
Chapter  V. 

C.  AREAS  FOR  FURTHER  RESEARCH 

(1)  The  implications  of  the  model's  "vertical”  and  "hori¬ 
zontal"  decompositions  should  be  further  explored.  That 
includes  modifications  suggested  by  the  thesis  such  as: 

(a)  Presenting  only  the  upper  level  value-dimensions 
to  the  decision-maker. 

(b)  Reviewing  issues  and  making  sub-decisions  referring 
the  separate  branches  of  the  model's  value-tree  and  only  as 

a  later  step  integrating  them  into  the  overall  model. 

(2)  The  approaches  and  methodologies  for  constructing 
utility  functions  for  major  systems  should  be  further 
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researched.  Some  of  them  may  have  general  application,  while 
others,  presumably,  are  unique  to  the  system  under  question. 

(3)  The  specific  procedural  and  organizational  adjustments 
required  for  MAUT  implementations  should  be  further  examined. 
That  should  be  the  duty  of  the  authorities  actually  responsi¬ 
ble  for  the  major  systems  acquisition  decision-making  process. 
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A- I .  Introduction  to  the  Questionnaire 

A.  GENERAL 

As  a  part  of  the  research  work  for  this  thesis,  a  question¬ 
naire  was  distributed  to  various  high-level  officials  in  the 
U.S.  DOD,  and  in  the  Israeli  Air  Force  ( IAF) . 

B.  PURPOSE 

The  purpose  of  the  questionnaire  was  to  obtain  inputs  asso¬ 
ciated  with  the  thesis  theme  from  the  people  who  are  actually 
and  personally  involved  in  the  major  systems  acquisition 
decision-making . 

There  were  two  main  reasons  why  two  decision-making  com¬ 
munities — the  American  and  the  Israeli — were  addressed: 

(1)  It  provided  the  basis  for  comparison  between  two  different 
disciplines,  dealing  with  the  same  issue,  while  one  is  not 
bound  by  the  perceptions,  tradition  and  procedures  of  the  other. 

(2)  The  special  interest  of  the  author  in  both  communities  as 
being  an  Israeli  officer  and  currently  a  student  at  the  Naval 
Postgraduate  School. 

C.  THE  QUESTIONNAIRE  OBJECTIVES 

The  main  objectives  of  the  questionnaire  were: 

(1)  To  examine  the  current  practices  of  top-level  acquisi¬ 
tion  decision-making  processes  and  find  out  to  what  extent 
quantitative  decision  models  are  in  use,  and  whether  a  further 
use  of  such  models  might  be  helpful. 

(2)  To  find  out  the  respondents  attitude  toward  a  specific 
model  (or  rather,  a  procedure)  based  on  the  Multi-Attribute 
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Utility  Technique  (MAUT)^,  and  its  compatibility  to  assist 
top-level  acquisition  decision-making. 

(3)  To  explore  the  respondents  ideas  about  the  actual 
implementation  of  the  above  mentioned  procedure  in  the 
decision-making  process. 

(4)  To  examine  the  possibility  of  constructing  a  general 

"value  tree",  i.e.,  to  define  value  dimensions  (or,  attributes, 

factors)  and  arrange  them  in  hierarchical  structure,  which, 

with  some  inevitable  modifications  would  apply  to  any  major 

2 

system  acquisition  decision-making  process. 

D.  THE  RESPONDENTS  POPULATION 
1 .  General 

As  mentioned  before,  the  questionnaire  was  distributed 
to  people  from  two  decision-making  communities:  the  U.S.  DOD 
and  the  Israeli  Air  Force.  In  both  cases  no  purely  random 
selection  of  the  sample  respondents  took  place.  They  were 
arbitrarily  selected  according  to  their  positions  and  estimated 
involvement  in  the  "real  life"  acquisition  decision-making 
process.  On  the  other  hand,  the  sample  was  selected  from  the 
DOD  Phone  Book  and  the  author  had  no  prior  idea  about  the 


^Also  called  in  its  various  versions  Multi-Attribute  Utility 
Theory  (MAUT) ,  Multi-Attribute  Utility  Measurement  (MAUM) ,  Multi- 
Attribute  Utility  Technology  (MAUT) ,  Multi-Attribute  Utility 
Analysis  (MAUA) ,  or  Simple  Multi-Attribute  Rating  Technique  (SMART) . 

2 

Originally,  the  questionnaire  was  supposed  to  concentrate 
mainly  on  fighter  aircraft  acquisition  decisions.  Later  on, 
the  scope  of  the  thesis  has  been  broadened  to  apply  to  more 
general  cases.  The  questionnaire  well  served  this  broader 
scope  as  well. 
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respondents  attitudes,  and  in  this  sense  there  was  some  extent 
of  "randomality"  in  the  selection,  as  reflected  in  the  devia¬ 
tion  of  the  answers. 

2 .  The  American  Distribution 

a.  The  questionnaire  was  distributed  to  73  American 

officials  of  which  42  have  responded  (57.5%).  4  claimed  they 

did  not  feel  eligible  to  fill  in  the  questionnaire  ("new  in 
the  job,"  "unfamiliar  with  the  subject,"  etc.).  Thus  38  filled 
in  questionnaires  came  back,  which  accounted  for  52%  of  the 
initial  distribution.  This  can  be  considered  as  an  excellent 
rate  of  response,  taking  into  account  that  the  questionnaire 
answers  required  about  2-3  working  hours  from  very  busy  high- 
level  personnel. 

b.  The  American  respondents'  population  accounted  for 
28  military  personnel  and  10  civilians.  The  ranks  of  the 
military  officers  were  distributed  as  follows: 


(1) 

LT.  GEN 

— 

2 

(2) 

V.  ADM. 

— 

4 

(3) 

MAJ.  GEN. 

— 

1 

(4) 

R.  ADM. 

— 

7 

(5) 

BRIG.  GEN. 

— 

1 

(6) 

CAPT  (Navy) 

— 

5 

(7) 

COL 

— 

4 

(8) 

LT.  COL 

— 

3 

(9) 

LT.  CDR 

— 

1 

10  American  civilians  responded  to  the  questionnaire.  The 
civilians  were  all  of  relatively  high  levels,  from  GS-15, 
SES  IV,  and  above. 


c.  The  respondents,  spread  according  to  their  DOD 


components , 

were  as  follows: 

(1) 

OSD 

—  13 

(2) 

Navy 

Department 

—  16 

(3) 

Air 

Force  Department 

9 

d.  The  respondents  were  from  a  broad  range  of  posi¬ 
tions  in  the  OSD  and  the  Service  Departments,  starting  at  the 
level  of  staff  members,  executive  assistants  and  program 
managers,  and  up  to  the  level  of  Under  Secretary  of  Defense. 
The  common  denominator  for  all  respondents  was  involvement  in 
the  major  systems  acquisition  decision-making  process,  within 
the  OSD  and  the  Services . 

e.  The  Israeli  respondents  population  consisted  of  11 
officers  of  the  IAF .  Their  ranks  were  distributed  as  follows: 


(1) 

BRIG.  GEN 

4 

(2) 

COL 

— 

3 

(3) 

LT.  COL 

— 

2 

(4) 

MAJ. 

— 

2 

Having 

in  mind  that  the 

rank  of  the 

IAF  Chief  of 

Staff  is  "only 

MAJ.  GEN.,  the  level  of 

the 

Israeli 

respondents 

were  considered 

as  equivalent  to  the  Americans.  Thus,  the  Israeli  respondents 
population  included  two  ex-deputies  to  the  IAF  Chief  of  Staff 
(the  second  position  in  the  IAF  hierarchy) ,  the  IAF  Chief  of 
Maintenance  and  Logistics,  IAF  HQs  department  heads,  etc. 


139 


A-II.  Example  of  the  Questionnaire 


A .  GENERAL 

This  appendix  presents  the  American  version  of  the  ques¬ 
tionnaire,  with  the  escort  letters  attached  to  it.  As  one 
can  notice,  the  title  of  the  questionnaire  speaks  about 
"Methodology  for  Aircraft  Acquisition  Decision-Making."  This 
is  not  identical  to  the  thesis  title.  The  reason  for  this 
discrepancy  is  that  according  to  the  material  accumulated 
through  the  research  for  this  thesis,  the  author  has  decided 
to  broaden  the  scope  of  the  thesis  theme.  Nevertheless,  this 
change  does  not  affect  the  substantial  application  of  the  ques¬ 
tionnaire  in  providing  insight  into  the  top-level  acquisition 
decision-making.  The  Israeli  version  of  the  questionnaire  is 
not  introduced  in  this  context  since  it  is  similar  to  the 
American  one.  On  the  other  hand,  because  of  some  natural 
differences  between  the  two  responding  populations,  and  because 
both  questionnaires  were  not  distributed  in  the  same  time-frame, 
there  were  some  substance  and  format  differences,  especially 
in  Part  V,  which  made  the  answers'  consolidation  sometimes 
difficult.  The  differences  are  addressed  specifically  where 
necessary . 

B.  THE  QUESTIONNAIRE 
Dear  Sir: 

First,  I  would  like  to  introduce  myself.  I  am  Colonel  Ran 
Goren,  a  pilot  in  the  Israeli  Air  Force.  I  have  about  4000  jet 
flight  hours,  and  250  operational  sorties.  I  did  a  variety  of 
jobs  at  the  field  and  Headquarters,  among  them  squadron  com¬ 
mander,  vice  wing  commander,  and  head  of  a  department  in  the 
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ISAF  Headquarters.  Currently,  I  am  a  student  at  the  Naval 
Postgraduate  School,  Monterey,  California. 

As  a  requirement  for  the  master's  degree,  I  have  to  submit 
a  thesis.  As  a  theme  for  my  thesis,  I  have  chosen  "The  Method¬ 
ology  of  Decision-Making  in  Aircraft  Acquisition."  The  com¬ 
pleted  thesis  will  be  available  to  the  DoD  acquisition  com¬ 
munity  through  the  NPS  and  the  Naval  Center  for  Acquisition 
Research. 

I  am  definitely  aware  of  how  big  and  complex  the  issue  is 
and  therefore,  I  have  limited  myself  as  follows: 

1.  To  deal  only  with  the  top-levels  of  decision-making 
(in  American  terms — (S)SARC,  (S)Sec,  DSARC  and  SecDef — or 
equivalent  to  those) . 

2.  Not  to  deal  in  detail  with  the  whole  acquisition  life 
cycle  process.  The  process  would  be  referred  to  only  in  terms 
of  the  inputs  it  should  provide  for  the  milestone  decision¬ 
making. 

3.  To  analyze  mainly  milestones  I  (demonstration/validation) 
and  II  (full  scale  development) . 

I  thought  that  getting  the  personal  perception  of  people 
involved  in  this  decision-making  might  be  a  vital  contribution 
to  my  work.  In  order  to  obtain  this  perception,  I  am  dis¬ 
tributing  the  enclosed  questionnaire. 

Although  the  questionnaire  is  voluntary,  I  would  like  you 
very  much  to  respond. 

If  you  find  it  too  long,  please  answer  personally  the 
first  half  (Parts  I,  II,  III)  only. 

Since  the  distribution  of  the  questionnaire  is  very  small, 
it  will  be  analyzed  by  simple  percentages,  and  ANOVA  (Analysis 
of  Variance) . 

I  am  not  going  to  quote  any  respondent  by  name.  Names  and 
positions  are  needed  just  for  relating  the  responses  to  the 
respondents  viewpoints.  But,  if  you  prefer  not  to  fill  in 
your  personal  information — please  leave  it  blank  or  put  just  an 
indication  of  your  level. 

I  will  be  grateful  if  you  return  your  response  up  to  the 
end  of  July.  At  any  rate,  after  August  31st,  the  questionnaire 
will  be  longer  be  administered.  Please  return  the  completed 
questionnaire  form  to  the  following  address: 

Colonel  Ran  Goren 
Naval  Postgraduate  School 
SMC  #1183 

Monterey,  CA  93940 
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Thanks  a  lot  for  your  cooperation. 


Sincerely  yours, 


Dear  Sir: 

As  you  may  know,  I  recently  moved  from  being  the  Principal 
Deputy  Under  Secretary  of  Defense  to  Dr.  W.  Perry  to  being  an 
Adjunct  Professor  here  at  the  Naval  Postgraduate  School.  As 
part  of  this  way  for  me  to  finish  the  last  few  months  of  my 
civil  service,  I  have  been  writing  down,  for  the  new  Under 
Secretary  Dick  De  La uer,  things  which  may  otherwise  be  lost 
in  the  transition.  One  of  these  areas  of  interest  to  both 
Dr.  De  Lauer  and  myself  has  been  how  best  to  improve  the 
acquisition  process. 

The  purpose  of  this  note  is  to  introduce  the  request  made 
of  you  to  assist  in  an  important  study  of  one  facet  of  our 
acquisition  process  being  done  by  Colonel  Ran  Goren,  IAF,  as 
part  of  his  thesis  work  at  the  Naval  Postgraduate  School, 
Monterey,  California. 

The  study  will  survey  an  important  aspect  of  the  acquisition 
process,  namely  whether  there  exists  an  adequate  review  of 
alternatives  before  selection  of  hardware  for  development  or 
acquisition.  Colonel  Goren  will  also  present,  for  comment  a 
potential  qualitative  technique  for  aiding  in  this  selection. 

His  thesis,  when  published,  will  be  available  to  you  and 
to  others  in  the  acquisition  community.  It  should  provide 
an  especially  pertinent  insight  into  an  important  acquisition 
question.  It  is  for  that  reason  I  ask,  as  a  favor,  your  con¬ 
sideration  of  his  request  to  put  down  your  thoughts  through 
the  media  of  his  questionnaire. 


W.  B.  LaBerge 
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A  Questionnaire 

Methodology  for  Aircraft  Acquisition  Decision-Making 


Respondent  Information: 


rank  first  name  middle.  last 


Address 


Present  Job 


Main  Past  Jobs  Connected  With  Acquisition 

*  • 


2. 


Preface 


Part  IV  of  the  Decision  Coordination  Paper  (DCP)  requires: 

"Summarize  system  and  program  alternatives  considered 

and  the  reasons  why  the  preferred  alternative  was  selected." 

Part  VI  of  the  DCP  requires:  "Identify  and  assess  issues 
affecting  the  SecDef's  milestone  decision,"  which  presuma¬ 
bly  are  political  and  socioeconomic  issues. 

The  DCP  is  supplemented  by  a  list  of  various  goals  and 
thresholds,  costs  and  resources. 

The  Integrated  Program  Summary  (IPS)  addresses  2_3  different 
topics . 

There  can  be  defined  1_6  decision  criteria  for  milestone  I 
and  II  decisions. 

The  AF  (as  an  example)  requires  about  35^  decision  data 
topics  to  base  on  in  its  milestone  review. 


The  milestone  decisions  might  be  a  very  complex  task,  even  after 
the  analyses,  data  collection,  refinements  and  eliminations 
that  have  been  done  through  the  preceding  process. 

This  questionnaire  attempts  to  explore  some  aspects  of  the  above 
problem  as  background  for  the  thesis  decision  model  development. 


3 .  General  Instructions 

a.  Ignore  questions  which  do  not  apply  to  you. 

b.  While  answering  the  question  refer  as  much  as  possible  to 
cases  of: 

1)  Milestones  I  and  II  decisions. 

2)  Selection  between  two  or  more  different  types  of 
weapon-systems  (aircralt  type  A  vs.  aircraft  type 

B,  and  not  a  selection  between  alternative  acquisition 
strategies  of  tb.2  same  system)  . 

3)  If  possible,  refer  to  aircraft  or  airborn  weapon 
systems  (although  other  systems  are  ok  too) . 

c.  Although  the  questions  aim  at  top-level  decision-making, 
you  may  use  your  experience  in  other  levels,  as  long  as 
multicriteria  selection  between  acquisition  alternatives 
is  involved. 
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Part  I 


Instructions : 

For  each  question  select  one  answer.  Please,  circle  the  number 
to  the  left  of  the  selected  answer. 


1.  Are  the  evaluation  of  alternatives  and  the  explanation  for 
recommendations  in  the  DCP  made  in  a  sufficiently  descrip¬ 
tive  manner  to  be  useful? 


2. 


1.  Never 

2.  Seldom 

3.  Sometimes 

4 .  Usually 

5 .  Always 


Are  the  evaluation 
recommendations  in 
tative  model? 


of  alternatives  and  the  explanation  for 
the  DCP  supported  by  sort  of  a  quanti- 


1.  Never 

2 .  Seldom 

3 .  Sometimes 

4.  Usually 

5 .  Always 


3.  In  the  milestones  review  sessions  (DSARC,  (S)SARC) — how 
often  is  the  evaluation  of  alternatives  supported  by  a 
normative — quantitative  model? 


1 .  Never 

2 .  Seldom 

3 .  Sometimes 

4.  Usually 

5.  Always 


4.  If  you  ever  used  a  model  for  overal  evaluation  of  acquisi¬ 
tion  alternatives  (not  necessarily  at  milestones  decisions) , 
which  one  did  you  find  the  most  useful? 

1.  Scores  and  weights  assigned  to  factors 

2.  A  decision  tree 

3.  A  computerized  model 

4.  Another  (which  one?) 

5.  Never  used  a  model 
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Part  II 


Instructions: 

In  each  question  there  is  a  statement.  For  each  statement 

select  your  degree  of  agreement. 

5.  There  is  no  problem  in  selection  of  an  alternative  at  the 
milestones  reviews,  since  the  pre-DSARC  process  leads  to 
refined,  clear  and  explicit  recommendations. 

1.  Decidedly  agree 

2.  Moderately  agree 

3.  Perhaps  agree 

4.  Moderately  disagree 

5.  Decidedly  disagree 

6.  After  hearing  (or  reading)  the  detailed  evaluation  of  the 
alternatives  by  individual  areas  (i.e.,  cost,  performance, 
etc.),  it  is  difficult  to  determine  which  alternative 

was  preferred. 

1.  Decidedly  agree 

2.  Moderately  agree 

3.  Perhaps  agree 

4.  Moderately  disagree 

5.  Decidedly  disagree 

7.  The  less  the  selection  is  clear  cut,  the  greater  is  the 
decision-maker's  tendency  to  rely  on  lower  level  analysis 
and  recommendations . 

1.  Decidedly  agree 

2 .  Moderately  agree 

3.  Perhaps  agree 

4.  Moderately  disagree 

5.  Decidedly  disagree 

8.  in  cases  where  the  overall  preference  is  unclear,  a  quanti¬ 
tative  model  might  be  helpful. 

1.  Decidedly  agree 

2.  Moderately  agree 

3.  Perhaps  agree 

4.  Moderately  disagree 

5.  Decidedly  disagree 

9.  In  fact,  top  level  decision-makers  do  not  select  between 
alternatives,  but  approve  or  disapprove  the  recommended 
alternative. 

1.  Decidedly  agree 

2.  Moderately  agree 

3.  Perhaps  agree 

4.  Moderately  disagree 

5.  Decidedly  disagree 
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10.  There  is  no  use  for  a  decision-making  model  to  support  top 
level  decision-makers  because  they  base  their  judgement 
mainly  on  a  subjective  experience,  perception  and  intuition 
which  cannot  be  rationally  "managed." 

1.  Decidedly  agree 

2.  Moderately  agree 

3.  Perhaps  agree 

4.  Moderately  disagree 

5.  Decidedly  disagree 


Part  III 
Instructions: 

Recall  2  cases  of  aircraft  acquisition.  (If  you  wish  you  may 
relate  to  systems  other  than  aircraft,  or  even  to  an  unspeci¬ 
fied  general  case.  Please,  note  what  kind  of  a  system  you 
are  relating  to.)  Evaluate  the  relative  weights  of  three 
main  categories; — direct,  socioeconomical  and  political  factors 
in  the  evaluation.  Express  weights  by  percentage.  Please, 
make  different  evaluations  for  some  different  levels. 


Case  1  Case  2 


Level  of  decision 

Dir. 

SE. 

Polit . 

Dir. 

SE. 

Polit. 

(Example) 

35% 

15% 

50% 

60% 

20% 

20% 

SecDef 


DSARC 
(S) Sec. 
(S) SARC 
Below 


147 


Part  IV 


Instructions : 

This  part  contains  a  simple  evaluating  model  example.  Ques¬ 
tions  in  the  following  parts  of  the  questionnaire  refer  to 
this  example.  The  example  is  a  submodel  of  an  overall  model, 
which  evaluates  the  alternatives  in  one  category  (change  in 
the  Air-Force  capability) .  Scores  assigned  relative  to  a 
specified  benchmark  (i.e.,  acceptable  criteria,  threshold) 
according  to  a  key: 

0  -  under  threshold  (or  minimal  criteria) . 

1  -  poor 

2  -  fair 

3  -  good 

4  -  very  good 

5  -  excellent 

Significance  weights  express  the  impact  of  the  factor  on  the 
evaluation  within  the  category  (they  should  sum  up  to  1) . 


Change  in  the  Air-Force  Capability  -  Example  Model 


Evaluation  Factor 


i  Score  i  Weighted  Score 

Weight  "  AltT  1  l  Alt:  2  '  Alt?  1~ '  Alt .  2 


Effectiveness  (with  j 
respect  to  MENS) _ j_ 


30 


1.5 


Cost-Effectiveness 


25 


.  D 


Timeliness 


15 


.45 


1 

Manpower  and 

Training 

.15 

i 

i  3  ! 

1 

i 

2  '  .45 

'  .30 

Maintainability 
and  Reliability 

I  .10 

1  4  i 

l 

2  !  _.4 

I-  -•  2_  ,  : 

RSI 

.  05 

2 

1 

!  ..  4  ,  4 

!  .2 

Overall  evaluation 


3.4' 


.  9 


1.25 


.15 


3.00 


Alternative  1  is  preferred. 


Comment:  Scores  and  weights  are  arbitrarily  assigned.  I  refer 

to  the  model  to  illustrate  the  principle  ideas  and  not  to  the 
specific  selection  of  factors,  weights  and  scores. 
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Part  V 


Instructions : 

This  part  contains  a  list  of  statements  with  respect  to  using 
such  a  model.  Please,  indicate  your  degree  of  agreraeent  to 
the  statements  by  circling  a  number  according  to  the  following 
key: 

Decidedly  agree  -  1 

Moderately  agree  -  2 
Perhaps  agree  -  3 

Moderately  disagree  -  4 
Decidedly  disagree  -  5 

11.  Such  a  model  may  be  used  only  if  supplemented  by  represen¬ 
tation  of  the  detailed  data  for  each  topic. 

1  2  3  4  5 

12.  The  decision-maker  himself  has  to  determine  weights  and  scores. 

1  2  3  4  5 

13.  The  model  is  supposed  to  be  objective  therefore  may  not  be 
changed.  (The  decision-maker  is  not  bound,  of  course, 

by  the  model  outcomes) 

1  '  2  3  4  5 

14.  The  model  is  prepared  by  an  analytical  process  through  the 
different  levels,  but  the  decision-maker  may  change  weights 
and  scores  (interact  with  the  model)  according  to  his 
judgement: 

1  2  3  4  5 

15.  Such  a  model  is  not  for  acquisition  related  decisions. 

1  2  3  4  5 

16.  Such  a  model  may  be  useful  in  most  cases  of  acquisition 
selection  decisions. 

1  2  3  4  5 

17.  Such  a  model  is  not  useful  for  top-level  decision-making. 

1  2  3  4  5 

18.  Such  a  model  is  useful  for  all  levels  except  SecDef,  since 
political  aspects  can't  be  handled  by  the  model. 

1  2  3  4  5 

19.  Evaluation  by  separate  factors  is  not  appropriate,  since 
the  appropriate  evaluation  is  derived  from  the  complex 
interrelations  among  factors. 

4 


1 


2 


3 
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5 


f 

Key 

Decidedly  agree  -  1 

Moderately  agree  -  2 

Perhaps  agree  -  3 

Moderately  disagree  -  4 

i  Decidedly  disagree  -  5 

20.  The  model  leads  people  to  manage  their  judgement  in  a 
logical  manner,  which  helps  the  decision-making. 

1  2  3  4  5 

21.  Such  a  model  leads  people  to  refer  to  the  really  important 
issues  in  the  decision. 

1  2  3  4  5 

22.  Such  a  model  suppresses  the  natural  intuitive  judgement. 

1  2  3  4  5 

23.  Such  a  model  may  bias  the  decision-makers  good  judgement. 

1  2  3  4  5 

24.  Such  a  model  may  somehow  bind  the  decision-makers  freedom. 

1  2  3  4  5 

25.  Such  a  model  enables  better  communication  among  people 
involved  in  the  decision-making. 

1  2  3  4  5 

26.  Such  a  model  may  be  used  as  a  tool  to  criticize  the 
decision  (e.g. ,  by  Congress),  and  therefore  undesirable. 

1  2  3  4  5 

27.  Such  a  model  is  a  very  good  way  to  reach  a  decision  in  a 
multicriteria  case. 

1  2  3  4  5 

28.  There  is  no  need  for  weights  and  total  score.  A  score  for 
each  factor  separately  is  sufficient.  The  rest  should  be 
left  for  pure  judgement. 

1  2  3  4  5 

29.  Such  a  model  may  shift  the  discussion  from  the  significant 
issues  to  unnecessary  arguments  on  scores  and  weights. 

1  2  3  4  5 

30.  Weights  and  scores  should  be  assigned  independently  from 

each  other  at  a  different  time,  and  perhaps,  by  different  people. 

1  2  3  4  5 
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Key 

Decidedly  agree  -  1 

Moderately  agree  -  2 

Perhaps  agree  -  3 

Moderately  disagree  -  4 

Decidedly  disagree  -  5 

31.  The  weights  are  the  main  ingredient  in  the  model  outcome, 
therefore  they  should  be  determined  by  the  decision-maker 
himself . 

1  2  3  4  5 

32.  Weights  and  scores  should  be  determined  by  a  common  work 
of  a  group  of  experts. 

1  2  3  4  5 

33.  Weights  and  scores  should  be  determined  by  summing  up 
assignments  made  individually  by  people  involved  in  the 
decision. 

1  2  3  4  5 

34.  Add  additional  remarks  with  respect  to  the  model. 
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Part  VI 


Instructions : 

Based  upon  your  knowledge  and  experience  of  two  recent  cases 
of  selection  between  acquisition  alternatives  -  evaluate  the 
following  list  of  possible  factors  which  affect  the  decision¬ 
making. 

a)  For  each  factor  assign  the  degree  of  impact  on  your 
evaluation  of  the  alternatives. 

b)  For  each  case  note: 

1)  the  nature  of  the  weapon  system  under  discussion 
(aircraft,  missile,  ship,  etc.) 

2)  the  phase  in  the  acquisition  life  cycle  you  refer  to. 
Rate  the  degree  of  impact  using  the  key: 


No  impact  -  0 
Weak  impact  -  1 
Medium  impact  -  2 
Strong  impact  -  3 


Very  strong  impact  -  4 


Case  1  Case  2 


Weapon  System 
Phase  of  Decision 


Factors  Impact  on  Evaluation 

1)  Effectiveness  (relative  to  MENS)  _  _ 

2)  Cost-effectiveness  _  _ 

3)  Timeliness  _  _ 

4)  RSI  _  _ 

5)  Maintainability  and  reliability  _  _ 

6)  Manpower  and  training  _  _ 

7)  Vulnerability  _  _ 

8)  Life  cycle  cost  _  _ 

9)  R&D  costs  _  _ 

10)  Production  costs  _  _ 
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No  impact  -  0 
Weak  impact  -  1 
Medium  impact  -  2 
Strong  impact  -  3 
Very  strong  impact  -  4 


Impact  on  Evaluation 


Factors 


Case  1 


Case  2 


11)  Initial  investment  _  _ 

(procurement  and  infrastructure) 

12)  Operating  and  support  costs  _  _ 

13)  Funding  resources  _  _ 

14)  State  of  the  art  _  _ 

(implemented  in  the  system) 

15)  R&D  current  status  _  _ 

16)  Feasibility  and  risks  _  _ 

17)  Critical  components  _  _ 

18)  Maturity  (experience  with  the  _  _ 

system) 

19)  Contractor  performance  _  _ 

(technological  achievements,  cost 

control,  timeliness) 

Please,  add  more  factors  that  you  think  deserve  to  be  evaluated 
at  high-levels  (except  political  and  socioeconomical) . 
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Part  VI 


Instructions ; 

In  this  part  I'll  appreciate  any  of  free  comments  with  respect 
to  the  questionnaire  subjects. 


End  of  the  questionnaire. 
Again,  thanks  for  your  cooperation. 


Ran  Goren 

N.P.G.S.  SMC  1183 
Monterey,  CA  93940 
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A- II I.  Introduction  to  the  Analysis  of  the  Questionnaire 


A.  GENERAL 

The  following  chapters  include  the  analysis  and  the  sum¬ 
maries  of  the  numerical  results  of  the  questionnaire.  It  is 
emphasized  that  the  analysis  in  these  chapters  refers  to  the 
questionnaire  only,  and  attempts  to  represent  objectively  the 
aggregate  attitudes  of  the  respondents.  Any  broader  view  of 
the  thesis  theme  based  on  additional  sources  is  left  for  the 
body  of  the  thesis. 

B.  SAMPLE  CHARACTERISTICS 

The  responding  sample  was  not  purely  randomly  selected. 
Thus,  from  a  pure  scientific  viewpoint,  the  results  and  the 
analysis  cannot  be  considered  as  an  unbiased  representation 
of  the  total  population  of  people  involved  in  top-level  acqui¬ 
sition  decision-making.  In  the  worst  case  the  results  may  be 
considered  as  representing  the  responding  sample  only.  But 
the  responding  sample  has  a  great  significance  in  its  own 
right.  As  mentioned  in  Appendix  A,  it  consists  of  38  DOD 
personnel,  28  military  and  10  civilians,  of  which  23  are  at 
high  ranks  (Generals,  Admirals  and  the  equivalent  civilian 
ranks),  up  to  USD  (R&E) .  It  represents,  by  more  or  less  equal 
portions,  the  OSD,  the  Navy  Department  and  the  Air  Force 
Department.  The  Israeli  responding  group  consists  of  11  Air 
Force  officers  including  4  Generals  of  the  highest  positions 
concerning  acquisition.  Consequently,  though  not  taking  for 
granted  any  statistical  result,  the  responding  sample  may  well 
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represent  opinions,  attitudes  and  trends  within  the  top-level 
acquisition  decision-making  communities  of  both  the  US  and 
Israel . 

C .  GENERAL  METHODOLOGY 

The  numerical  summaries  and  the  analysis  are  presented  in 
6  chapters  (IV  to  IX) .  In  each  chapter  the  numerical  summaries 
and  the  analysis  associated  with  the  subject  under  discussion 
are  attached  together  for  the  reader's  convenience.  The  sub¬ 
jects  of  the  chapters  follow  the  underlying  questionnaire  sur¬ 
veying  purposes,  but  not  necessarily  create  a  complete  logical 
continuity . 

The  questions  are  referred  to  in  these  chapters  according 
to  the  issues  they  inquire,  and  not  according  to  their  order 
of  appearance  in  the  original  questionnaire.  This  is  also  the 
rationale  of  the  numerical  summaries  presentation  order. 

The  subjects  of  the  chapters  are  as  follows: 

(1)  Examination  of  the  present  situation  in  top-level 
acquisition  decision-making. 

(2)  Competability  and  usefulness  of  quantitative  decision¬ 
making  techniques  for  top-level  acquisition  decision-making. 

(3)  Advantages  and  disadvantages  of  MAUT  as  a  top-level 
acquisition  decision-making  tool. 

(4)  Implementation  of  MAUT  procedure  in  major  system  acqui¬ 
sition  decision-making  process. 

(5)  Relative  importance  of  acquisition  decision-making 


attribute  categories. 


(6)  Identifying  acquisition  decision-making  attributes 
(value  dimensions) . 

The  responding  sample  is  referred  to  three  categories: 

(1)  Low  levels — Colonels,  Captains  (Navy),  GS-15,  and  below. 

(2)  High  levels — Generals,  Admirals  and  the  equivalent 
civilian  ranks. 

(3)  Total — all  levels  together. 

The  rationale  behind  this  division  rests  on  the  assumption 
that  the  "low"  levels  are  mostly  staff  members  while  the  "high” 
levels  are  personally  involved  in  the  decision-making.  Although 
such  distinction  is  oversimplified,  it  has  been  selected  for 
convenience  reasons,  under  the  assumption  that  no  major  error 
will  result. 

D.  METHODOLOGY  OF  THE  NUMERICAL  SUMMARIES 

The  numerical  summaries  are  done  on  a  single  question  basis, 
i.e.,  each  question  has  a  separate  summary.  Interrelations 
among  questions  are  addressed  in  the  verbal  analysis. 

As  a  result  of  the  small  sample  size,  no  complex  statistics 
are  developed.  Rather,  a  simple  frequency  distribution  is  the 
most  common  tool  for  this  analysis.  The  frequency  distribution 
is  introduced  in  three  shapes : 

(1)  Absolute  amount. 

(2)  Relative  frequency  (expressed  as  percentage) 

(3)  Relative  frequency  histogram. 

Some  differences  between  the  American  and  the  Israeli  ques¬ 
tionnaire  versions  do  not  allow  a  consolidation  of  the  data. 

As  a  matter  of  fact,  such  a  consolidation  is  not  desirable  since 
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a  comparable  analysis  is  preferred.  Thus,  the  Israeli  sum¬ 
maries  appear  separately.  Because  of  the  small  Israeli  sample 
size,  only  the  total  frequency  distribution  is  exhibited, 
although  the  data  about  the  separate  distributions  of  the 
Israeli  "high"  and  "low"  levels  does  exist,  and  it  is  referred 
to  in  the  discussion  of  the  analysis  when  necessary.  In  the 
Israeli  questionnaire  there  are  only  four  alternative  answers 
for  each  question,  compared  to  five  in  the  American  version. 

This  requires  some  adjustments  for  comparison  needs.  Part  V 
of  the  Israeli  questionnaire  is  structured  differently  from 
the  corresponding  part  in  the  American  version.  Therefore  the 
summary  of  this  part  for  the  Israeli  answers  only  is  located 
separately  at  the  end  of  the  associated  chapters. 

i 

E.  METHODOLOGY  OF  THE  ANALYSIS 

The  analysis  refers  to  the  questionnaire  answers  and  written 
comments  only.  Neither  the  analysis  of  the  questions,  nor  the 
chapters'  summaries  are  supposed  to  represent  the  final  con¬ 
clusions  of  this  thesis.  That  is  the  reason  why  there  are  no 

I 

conclusions  to  the  analysis.  The  conclusions  are  deliberately 
left  for  the  body  of  the  thesis. 

To  facilitate  the  analysis,  the  five  answer  alternatives  ^ 

are  usually  grouped  into  three  categories: 

(1)  The  "agree"  category  includes  the  "decidedly  agree" 
and  "moderately  agree"  answers. 

(2)  The  "disagree"  category  includes  the  "decidedly  dis¬ 
agree"  and  the  "moderately  disagree"  answers. 
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(3)  The  "middle"  category  includes  the  "perhaps  agree 


answers . 

The  division  is  equivalent  where  frequency  type  answers 
(never,  seldom,  etc.)  or  others  are  concerned.  Obviously, 
where  necessary,  the  full  fine  distinction  through  all  five 
alternative  answers  is  used. 

For  a  common  language,  four  types  of  distributions  are 
defined: 

(1)  The  "uniform"  distribution — the  answers  are  more  or 
less  uniformally  distributed  through  the  various  answer 
alternatives,  as  illustrated  in  Figure  A-III-1. 


Figure  A-III-1.  "Uniform"  Distribution 

(2)  The  "split"  distribution — the  answers  are  divided  cate¬ 
gorically  on  both  sides,  while  very  few  are  in  the  middle 


* 

In  this  and  all  future  exhibits,  the  following  abbreviations 
are  used  to  label  the  entries  along  the  x-axis:  D.A.  =  Decidedly 
Agree,  M.A.  *  Moderately  Agree,  P.A.  =  Perhaps  Agree,  M.D.  = 
Moderately  Disagree,  D.D.  =  Decidedly  Disagree.  See  above  sections 
of  this  appendix  for  a  fuller  discussion  of  the  categories. 


column.  That  means  that  people  have  clear  opinions  about  the 
subject,  and  they  are  either  pro  or  con.  Figure  A-III-2 
illustrates  this  distribution. 


Figure  A-III-2.  "Split"  Distribution 

(3)  The  "middle"  distribution— This  is  the  case  where  the 
largest  column  is  the  one  at  the  middle,  i.e.,  the  group  of 
people  who  do  not  have  definite  opinions  about  the  subject  is 
dominant.  Usually  the  two  next  in  size  are  the  columns  on  both 
sides  of  the  middle — the  "moderately  agree"  and  "moderately 
disagree"  columns.  Figure  A-III-3  illustrates: 
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Figure  A-III-3. 


Middle"  Distribution 


(4)  The  "one-sided"  distribution--The  answers  tend  to  be 
concentrated  on  one  side  of  the  histogram,  i.e.,  the  majority 
of  respondents  clearly  favor  one  attitude  to  the  subject. 
Figure  A-III-4  illustrates: 


Figure  A-III-4.  "One-sided"  Distribution 


Some  of  the  questions  appeared  in  different  shapes  in  the 


Israeli  questionnaire.  Instead  of  having  five  degrees  of 
agreement  for  each  question,  the  Israeli  respondents  were 
presented  a  list  of  statements  from  which  they  had  to  select 
only  those  which  they  think  are  true.  The  statements  them¬ 
selves  are  almost  identical  to  those  appearing  in  the  American 
version.  Thus,  the  numer  of  respondents  that  picked  up  a 
statement  is  the  number  of  those  who  agree  with  the  statement. 
Those  who  did  not  pick  up  a  specific  statement  probably  dis¬ 
agree  or  don't  have  a  determined  mind  about  it.  A  concentra¬ 
tion  of  that  type  of  q  stion  appears  at  the  end  of  each 
chapter,  consisting  of  the  questions  corresponding  to  the 
chapter's  theme. 

Generally,  each  question  is  treated  separately.  But 
where  necessary,  interrelations  among  questions,  mostly  within 
the  chapters,  are  addressed — verbally  and  statistically. 

The  American  questionnaire  serves  as  the  analysis  base, 
while  the  Israeli  one — since  quantitatively  small — is  used 
as  a  reference  and  comparison  mean. 

Another  notation  which  appears  in  this  analysis  relates 
to  the  quotes  from  the  written  comments  to  the  questionnaire. 

On  one  hand,  there  is  a  commitment  not  to  identify  commentator 
by  name.  On  the  other  hand,  it  is  important  to  recognize 
that  a  set  of,  say,  very  critical  comments,  came  from  the  same 
person,  and  they  do  not  necessarily  represent  a  wide  range  of 
respondents.  Therefore,  commentators  are  assigned  an  arbitrary 
notation  which  appears  in  brackets  following  the  quote  or 
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reference  citation, 
level  personnel  and 
?.  157)  as  follows: 
L-l,  L-2,  etc.,  for 


The  notation  distinguishes  between  high- 
low-level  personnel  (see  definition  on 
H-l ,  H-2 ,  etc.,  for  high-levels,  and 
low  levels. 


163 


A- IV .  Examination  of  the  Present  Situation 

A.  PURPOSE 

The  purpose  of  this  section  and  the  corresponding  ques¬ 
tions  is  to  reveal  what  are  the  real  difficulties  the  decision¬ 
makers  face  when  they  reach  a  decision  point. 

B.  DISCUSSION  ABOUT  THE  BASIC  APPROACH  OF  THE  ANALYSIS 

The  questionnaire  addresses  mostly  the  formal  decision 

sessions  and  documentation.  The  assumption  underlying  the 
above  approach  is  that  the  formal  decision  sessions  (e.g., 
Milestones  DSARCs)  or  documentation  (e.g.,  DCP)  do  reflect 
or  should  reflect  the  alternative  system  evaluating  techniques 
used  throughout  the  decision  process.  Obviously,  they  do  re¬ 
flect  the  decision  and  evaluation  tools  offered  to  the  decision¬ 
makers  themselves  for  their  decision-making. 

The  questions  deliberately  refrained  from  touching  the 
underlying  essential  acquisition  decision-making  difficulties, 
mainly  stemming  from  uncertainties,  such  as  how  to  make  a 
realistic  cost  estimate,  how  to  predict  the  performance  of  a 
system  that  has  not  been  born  yet,  or  how  to  define  the  real 
mission  needs,  based  on  a  projected  threat  assessment.  These 
warrant  a  treatment  beyond  the  scope  of  this  thesis.  However, 
the  assumption  is  that  better  evaluation  and  decision-making 
techniques  have  significant  impact  on  the  essence  of  the  deci¬ 
sion.  Several  commentators  write  that  dealing  mostly  with  the 
so-called  "DSARC  Process",  or  "A-109  process"  is  inappropriate 
since  it  is  subject  to  continuing  revisions,  including  expected 


near-term  substantial  change  [H-1,L-1] .  Others  mention  that 
the  "acquisition  process  documentation  is  of  limited  value, 
if  any..."  [H-l] .  They  emphasize  the  informal  decision  process 
that  takes  place  parallel  to  the  formal  one,  and  the  "less 
conspicuous  decision-makers,  who  are  often  more  influential 
than  the  formal  and  obvious  ones"  [ H— 2 ] . 

While  admitting  that  the  issue  is  obviously  more  complex 
than  the  questionnaire  will  permit,  several  points  can  be  made 
to  stress  the  value  of  the  data  that  this  part  of  the  ques¬ 
tionnaire  does  provide.  These  points  are: 

(1)  Decision-making  tools  are  not  attached  to  a  specific 
formal  decision  process.  Change  of  the  present  process,  say, 

to  a  greater  extent  of  decision  authority  granted  to  the  Services 
less  DSARC  Milestone  sessions,  or  more  pages  allowed  in  the 
DCP ,  does  not  necessarily  change  the  extent  to  which  decision¬ 
making  tools  are  used.  The  present  decision  difficulties  and 
decision-making  tool  usage  are  symptoms  of  basic  approaches 
and  not  attached  to  a  specific  decision  process. 

(2)  Correcting  the  weaknesses  of  the  formal  process  may  re¬ 
duce  the  informal  powers  affecting  the  process. 

(3)  The  formal  sessions  and  documentation  have,  after  all, 
a  significant  value.  For  very  busy  top-level  officials  who 
do  not  deal  with  the  acquisition  issues  on  a  day-to-day  basis, 
the  formal  sessions  and  the  documentation  they  receive  repre¬ 
sent  a  significant  part  of  the  input  for  their  decisions. 
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C.  DIFFICULTIES  IN  THE  DECISION  MAKING 


According  to  Q1  (see  Exhibit  1) ,  46%  of  the  American  re¬ 
spondents  think  that  the  evaluation  of  alternatives  and  the 
explanation  for  recommendations  in  the  DCP  are  made  in  a 
sufficiently  descriptive  manner  to  be  useful.  The  preference 
here  is  clear  as  compared  to  only  14%  who  think  that  it  is  so, 
seldom  or  never.  However,  no  less  impressive  is  the  percentage 
of  those  who  think  that  only  sometimes  are  the  descriptions 
in  the  DCP  good  enough  to  serve  as  a  decision  basis.  Thus,  a 
considerable  percentage  are  not  generally  satisfied  with  what 
is  presented  in  the  DCP.  Q6  (see  Exhibit  2)  reveals  very 
clearly,  with  69%  at  the  "disagree"  region  ("moderately  dis¬ 
agree"  plus  "decidedly  disagree") ,  that  after  the  alternatives 
demonstration  (presumably  with  the  current  techniques) ,  there 
is  no  difficulty  in  determining  which  alternative  was  preferred. 
This  question  might  have  been  misinterpreted  because  of  inex¬ 
plicit  wording.  The  question  was  aimed  at  the  decision-maker's 
difficulties  in  determining  his  own  preferences  based  on  the 
alternatives  presented,  and  not  on  his  ability  to  recognize  which 
alternative  was  preferred  by  the  program  sponsor.  This  possi¬ 
ble  confusion  is,  perhaps,  disclosed  by  the  corresponding  Q5 
(see  Exhibit  3)  which  deals  with  the  same  issue  of  preference 
or  selection  of  alternatives.  Here,  47%  of  the  respondents 
think  that  a  problem  does  exist  in  the  selection  of  alterna¬ 
tives  while  29%  think  that  there  is  no  problem,  and  24%  are 
inbetween.  This  is  supported  by  the  Israelis,  who  express  their 
reservation  with  64%  of  "Sometimes  true"  answers. 
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Exhibit  1 


Question  No.  1  (Ql) — Numerical  Summary 
The  Question's  Phrasing: 

Are  the  evaluation  of  alternatives  and  the  explanation 
for  recommendations  in  the  DCP  made  in  a  sufficiently 
descriptive  manner  to  be  useful? 

Table  A- IV- 1 

American  Responses  Distribution 


Never 

Seldom 

Sometimes 

Usually 

Always 


Distribution  (Amount 


Low  !  Hiah  I  Total 


Distribution  (Percent) 


Low 

mmm 

Tota: 

1 

7.1  i 

—  1 

I  2.7 

14.3  j 

8.7 

10.8 

28.6 

47.8 

40.5 

50.0 

32.5  j 

45.9 

- 

—  1 

— 

Never  Seldom  Some-  Usually  Always 
times 

Figure  A-IV-1.  American  Responses  Distribution  Histogram 


Table  A- IV- 2 

Israeli  Responses  Distribution 


Distribution  (Amount) 


Distribution  (Percent 


12.5 
25.0 

62.5 
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Question  No.  6  (Q6) — Numerical  Summary 
The  Question's  Phrasing: 

After  hearing  {or  reading)  the  detailed  evaluation  of  the 
alternatives  by  individual  areas  (i.e.,  cost,  performance, 
etc.),  it  is  difficult  to  determine  which  alternative 
was  preferred. 

Table  A- IV- 3 

American  Responses  Distribution 


Exhibit  3 


Question  No.  5  (Q5) — Numerical  Summary 
The  Question's  Phrasing: 

There  is  no  problem  in  selection  of  an  alternative  at  the 
milestones  reviews,  since  the  pre-DSARC  process  leads  to 
refined,  clear  and  explicit  recommendations. 

Table  A- IV- 5 

American  Responses  Distribution 


D.A.  M.A. 


Figure  A-IV-3.  Ameri 


Israeli  Resp 


Distribution  (Amount)  (  Distribution  (Percent) 


Decidedly  Agree 
Moderately  Agree 
Perhaps  Agree 
!  Decidedly  Disa 
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Several  respondents '  written  comments  disclose  a  new  factor 
which  on  the  surface  makes  the  selection  easier,  but  not  as  a 
result  of  objective  alternatives  presented  or  usage  of  effi¬ 
cient  decision-making  tools.  The  following  quotes  express 
these  observations: 

. . .The  real  alternatives  generally  do  not  appear — 
only  strawmen...  [H-3] 

...By  this  time  the  deck  is  so  stacked  that  the  viable 
alternatives  either  do  not  appear  or  have  had  a 
hatchet  job  done  on  them.  [H-3] 

...The  selection  is  never  less  than  clear-cut... 
the  Service  assures  this...  [H-3] 

...The  arguments  and  evidence  are  usually  structured 
to  support  the  Service  selection  and  to  present  the 
decision  in  the  most  "clearcut"  form  permitted  by  the 
facts...  [H-4] 

...By  the  time  such  reviews  are  held  at  the  highest 
levels,  for  all  practical  purposes  the  selection  has 
been  made,  and  any  other  choice  or  selection  is  very 
difficult  to  affect...  [H-4] 

Thus,  even  if  these  comments  should  be  taken  with  some 
reservations,  they  can't  be  ignored.  The  question  to  be  asked 
is  how  to  make  decision  sessions  more  effective  and  system 
presentation  more  objective.  This  question  is  addressed  in  the 
body  of  the  thesis,  at  least  as  far  as  decision-making  tech¬ 
niques  are  concerned. 

Another  group  of  difficulties  in  decision-making  is  re¬ 
vealed  by  Q7  (see  Exhibit  4) .  The  answers  distribution  is  of 
the  "split"  type,  where  people  either  agree  or  disagree, 
while  very  few  are  in  the  middle.  54%  disagree  with  the  state¬ 
ment  that  "the  less  the  selection  is  clear-cut,  the  greater 
is  the  decision-makers  tendency  to  rely  on  lower  level  analysis 


and  recommendations."  22% — a  relatively  high  amount  for  an 
extreme  answer — even  decidedly  disagree.  But  yet,  a  signifi¬ 
cant  portion,  35%  think  on  the  contrary. 

It  is  interesting  in  this  case  to  observe  the  differences 
between  high  and  low  levels:  while  high  levels  apparently  re¬ 
ject  the  statement  with  70%  disagreeing  and  only  26%  agree  (the 
remaining  4%  are  inbetween) ,  low  levels  agree  by  51%  and  only 
28%  disagree.  Whose  observation  is  more  accurate  is  hard  to 
tell.  The  Israeli  rejection  of  the  statement  is  more  definite, 
with  80%  in  the  "disagree"  category,  with  a  similar  distribu¬ 
tion  for  both  high  and  low  levels.  Perhaps  that  points  out 
a  higher  confidence  in  their  ability  to  make  a  decision  in  a 
difficult  selection.  This  feeling  gets  support  from  Q9  (see 
Exhibit  5) .  Here  54%  of  the  American  respondents  agree  that 
"in  fact,  top-level  decision-makers  do  not  select  between 
alternatives,  rather,  approve  or  disapprove  the  recommended 
one."  35%  objected  to  the  Q9  statement.  Furthermore,  among 
the  high  levels,  the  degree  of  agreement  is  even  higher  than 
the  total,  with  65%  that  agree!  On  the  Israeli  side  the  dis¬ 
agreement  is  obvious  with  73%  disagreeing.  Moreover,  all  four 
Israeli  Generals  decidedly  disagree  to  the  statement!  The 
reasons  for  the  American  observation  might  be  one  of  the  follow¬ 
ing  two,  or  both: 

(1)  The  top  level  decision-makers  do  not  have  sufficient 
tools,  data  or  expertise  to  allow  selection  between  alternatives. 

(2)  As  quoted  from  one  of  the  respondents:  "Alternatives 
other  than  the  preferred  alternative,  are  evaluated  to  gain 
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Exhibit  5 

Question  No.  9  (Q9) — Numerical  Summary 
The  Question's  Phrasing: 

In  fact,  top  level  decision-makers  do  not  select  between 
alternatives,  but  approve  or  disapprove  the  recommended 
alternative. 


Table  A-IV-9 

American  Responses  Distribution 


Distribution 

(Amount) 

(Percent) 

Total 

Lew 

Kiss 

Total 

Decidedly  Agree 

3 

7 

10 

23.8 

30.6 

27.0 

Moderately  Agree 

2 

8 

10 

14.3 

34.8 

27.0 

Perhaps  Agree 

2 

2 

4 

14.3 

8.7 

10.8 

Moderately  Disagree 

5 

4 

9 

35.7 

17.4 

24.3 

Decidedly  Disagree 

2 

2 

4 

14.3 

8.7 

10.8 

perspective  on  the  preferred  one.  There  is  virtually  no 
probability  any  other  will  be  selected"  [L-2] . 

D.  EXTENT  OF  QUANTITATIVE  MODELS  USAGE  IN  THE  PRESENT 

DECISION-MAKING  PROCESS 

The  purpose  of  Q2  (see  Exhibit  6)  was  to  reveal  the  extent 
quantitative  models  are  used  to  support  the  recommendation  in 
the  DCP.  The  major  observation  on  the  responses'  distribution 
is  the  high  amount  of  the  answer  "sometimes",  which  accounted 
for  38%.  The  "seldom"  and  "usually"  answers  share  about  an 
equal  percentage — 30%  and  2  7%,  respectively.  One  can  wonder 
why  there  is  any  distribution  at  all,  since  the  question  deals 
with  tangible  facts.  The  answer,  of  course,  is  that  none  of 
the  respondents  have  complete  knowledge  about  all  the  DCP's 
issued.  Thus  people  just  estimate.  In  this  respect,  it  is 
interesting  to  notice  that  perhaps  the  only  respondent  who  is 
connected  with  the  DCP's  on  a  day-to-day  basis,  and  reviews  all 
DCP  issues,  is  the  only  one  who  answers  "never",  with  the  simple 
verbal  explanation  that  a  "10  page  DCP  is  too  short  for  this" 
[L-3] .  Similarly  to  the  American  answers,  the  Israeli  response 
to  this  question  reveals  a  clear  majority  of  63%  who  estimate 
the  frequency  of  quantitative  model  use  in  presentations  to 
the  Minister  of  Defense  as  occurring  only  "sometimes". 

A  similar  picture  is  observed  in  the  answers  to  Q3  (see 
Exhibit  7)  which  asks  about  the  frequency  quantitative  models 
are  used  to  support  alternatives  evaluation  in  the  Milestones 
review  sessions  (DSARC,  (S)SARC).  Again,  there  is  a  "middle" 
distribution  with  the  largest  portion  selecting  the  "sometimes" 
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Exhibit  6 


Question  No.  2  (Q2) — Numerical  Summary 
The  Question's  Phrasing: 

Are  the  evaluation  of  alternatives  and  the  explanation 
for  recommendations  in  the  DCP  supported  by  sort  of  a 
quantitative  model? 

Table  A-IV-11 

American  Responses  Distribution 


Distribution  (Amount 


Distribution  (Percent)  i 


High 

!  Total  j 

4.3 

|  I 

1  5.4  | 

34.8 

!  29.7  j 

39.1 

37.8  | 

21.7 

27.0  | 

! 

Distribution  (Amount) 


Never 

Sometimes 

Usually 

Always 


Distribution  (Percent) j 


t 

62.5  i 

37.5  ! 
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Exhibit  7 


Question  No.  3  (Q3) — Numerical  Summary 
The  Question  Phrasing: 

In  the  milestones  review  sessions  ( DS ARC ,  (S)SARC) — how 
often  is  the  evaluation  of  alternatives  supported  by  a 
normative-quantitative  model? 

Table  A-IV-13 

American  Responses  Distribution 


Distribution 

(Amount) 

Distribution 

(Percent) 

Low 

Total 

Low 

High 

Total 

Never 

■■ 

2 

2 

9.1 

■HHi 

Seldom 

7 

12 

31.8 

i  Km  ■ 

Sometimes 

RS 

8 

14 

EH 

36.4 

38.9 

Usually 

5 

7 

" 

22.7 

19.4 

Always 

9HR 

- 

1 

ml 

- 

2.8 

times 

Figure  A-IV-7.  American  Responses  Distribution  Histogram 


Table  A- IV- 14 

Israeli  Responses  Distribution 


Distribution  (Amount) 

Distribution  (Percent) 

Never 

Sometimes 

Usually 

1  Always 

| 

1  ,9.1 

9  1  81.8 

1  !  9.1 
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answer.  But  here  the  tendency  toward  low  frequency  is  even 
stronger  than  in  Q2 ,  since  33%  of  the  answers  are  "seldom". 

At  least  two  commentators  devaluate  the  models  actual  utility 
not  only  because  of  the  low  frequency  that  they  are  used,  but 
also  because:  "the  model  is  always  severely  limited,  for  exam¬ 
ple,  it  might  use  acquisition  cost  as  the  only  basis  for  com¬ 
parison"  [H-3] ,  or,  "the  scope  of  the  models  that  usually 
"support"  the  recommendations  is  limited"  [H-4].  This  is 
strengthened  by  an  executive  who  is  supposed  to  attend  all  DSARC 
sessions,  who  mentions  that  the  only  models  he  has  seen  used  in 
these  sessions  were  those  of  the  Air  Force  briefings,  "however 
the  Air  Force  has  only  indicated  relative  comparisons  (using 
red,  green,  yellow)....  This  technique  has  met  with  mixed  suc¬ 
cess  since  sometimes  it  tends  to  hide  significant  issues"  [L-3] . 
Another  commentator  views  the  low  contribution  of  the  quantita¬ 
tive  models  not  only  because  of  their  limited  scope,  but  because 
the  forum  is  not  open  enough  to  exploit  what  they  do  offer. 

That  because  "the  review  sessions  are  never  so  emotionless  as 
to  deal  with  a  dispassionate  review  of  alternatives"  [H-3] . 

A  corresponding  question  to  the  Israeli  AF  officers,  deal¬ 
ing  with  sessions  equivalent  to  the  (S)SARC,  reveals  a  much 
greater  extent  of  quantitative  techniques  usage.  The  responses 
are  here  "usually"  (82%)  or  "always"  (9%) .  Maybe  this  is  the 
reason  for  the  confidence  in  their  ability  to  select  between 
alternatives,  not  only  to  approve  or  disapprove  the  recommended 
one,  as  their  American  colleagues  tend  to  do  (see  Q9 — Exhibit 
5)  . 
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E .  CHAPTER  SUMMARY 

From  the  responses  relating  to  the  subject  of  this  chapter, 
several  observations  can  be  drawn: 

(1)  In  many  of  the  cases  difficulty  exists  in  selecting 
an  alternative,  despite  the  refining  process  prior  to  top- 
level  decision-making. 

(2)  At  least  in  some  of  the  cases  the  selection  seems  to 
be  straightforward  because  of  biased  presentations  made  by 
the  Services  (or  other  program  sponsors) . 

(3)  American  top-level  decision-makers  tend  to  approve  or 
disapprove  the  recommended  alternative  rather  than  to  decide 
and  select  between  alternatives. 

(4)  The  usage  of  quantitative  models  or  techniques  to  as¬ 
sist  top-level  acquisition  decision-making  is  done  to  a  minor 
extent,  and  the  models  used  are  of  limited  scope. 

(5)  The  Israeli  decision-makers  tend  to  use,  to  a  greater 
extent,  quantitative  techniques,  and  that  is  expressed  also 
in  the  self-perception  of  their  decision-making  capabilities. 
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A-V.  Compatibility  and  Usefulness  of  Quantitative 
Decision-Making  Techniques 

A.  PURPOSE 

The  purpose  of  this  chapter  is  to  examine  the  respondents 
perceptions  about  the  compatibility  and  usefulness  of  quanti¬ 
tative  techniques  in  assisting  top-level  acquisition  decision¬ 
making.  A  special  attention  is  paid  to  a  MAUT  (Multi-Attribute 
Utility  Technique,  or  Theory) . 

B.  COMPATIBILITY  OF  QUANTITATIVE  TECHNIQUE^ — A  GENERAL 

PERCEPTION 

Q8  (see  Exhibit  8)  states  that  "in  cases  where  the  overall 
preference  is  unclear,  a  quantitative  model  might  be  helpful." 
One  can  interpret  this  question  as  relating  to  the  model  usage 
as  a  major  tool  for  overall  evaluation  of  weapon  systems  at 
any  level,  including  the  top-level,  or  as  a  mere  presentation 
tool  before  top-level  decision-makers.  In  fact,  there  is  a 
great  deal  of  similarity  between  the  two.  It  is  true,  and  it 
actually  happens,  that  models  are  used  in  lower  levels  but  are 
not  fully  presented  to  the  top-levels,  as  mentioned  by  one 
commentator : 

The  AFSARC  does  not  use  models  for  decision-making 
directly.  We  are,  however,  briefed  on  the  use  of 
decision  aids  and  quantitative  methods  used  by  the 
staff  in  developing  recommendations.  [H—  5 ] 

On  the  other  hand,  it  is  hard  to  imagine  a  model  used  only 
as  a  top-level  decision  aid  that  has  not  been  developed  and 
used  by  levels  below  them.  Thus,  Q8  refers  to  a  quantitative 
model  as  a  decision-making  aid  at  any  level,  including  the  top 
one.  The  distribution  of  Q8  answers  is  of  the  "middle"  type. 
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Exhibit  8 

Question  No.  8  (Q8) — Numerical  Summary 
The  Question's  Phrasing: 

In  cases  where  the  overall  preference  is  unclear,  a 
quantitative  model  might  be  helpful. 


Table  A-V-l 

American  Responses  Distribution 


Distribution 

(Amount) 

Distribution 

(Percent) 

Low 

High 

Total 

Low 

High 

Total 

Decidedly  Agree 

2 

2 

14.3 

- 

5.4 

Moderately  Agree 

3 

9 

12 

21.4 

39.1 

32.4 

Perhaps  Agree 

7 

11 

18 

50.0 

47.8 

48.6 

Moderately  Disagree 

2 

2 

4 

14.3 

8.7 

10. S  1 

Decidedly  Disagree 

- 

1 

1 

~~ 

4.3 

2.7  I 

100 


60 


Almost  50%  are  doubtful,  or  in  other  words,  "perhaps  agree" 
to  the  aid  a  quantitative  technique  might  provide.  But  the 
tendency  is,  after  all,  in  favor  of  the  model's  usefulness, 
since  among  those  who  are  more  determined,  38%  agree  with  the 
statement  and  only  14%  disagree.  The  Israeli  respondents  tend, 
in  greater  degree,  toward  usage  of  quantitative  techniques  as 
reflected  by  60%  that  agree  to  the  statement  and  40%  tend  to 
disagree  ("moderately  disagree") . 

A  corresponding  question,  Q10  (see  Exhibit  9) ,  concentrates 
more  specifically  on  top-level  decision-makers.  The  underlying 
rationale  to  the  question  is  that  one  might  think  that  quanti¬ 
tative  decision-making  techniques  can  be  compatible  mostly  to 
low-level  evaluations  and  decisions,  since  the  latter  are, 
perhaps,  more  technically  oriented  and  based  on  measurable  data. 
According  to  this  line  of  thought,  in  top-level  decision-making 
more  unmeasurable  value  dimensions  are  involved,  with  decisions 
based  mainly  on  subjective  judgment  or  intuition,  and  therefore 
do  not  lend  themselves  to  quantitative  decision-making  tools. 

This  idea  appears  in  some  of  the  written  comments  to  the 

questionnaire.  Several  quotes  are  provided  for  illustration: 

At  the  program  (office]  level  we  tend  to  do  little 
modeling  outside  the  engineering  domain,  where  we  get 
good  correlation  factors.  Objective  modeling  tends 
discipline  but  is  not  our  normal  tool  in  decision¬ 
making.  [L— 4  ] 

I  am  not  favorably  impressed  with  complex  numerical 
modeling  as  a  basis  for  important  weapon  systems  deci¬ 
sions.  A  clear  logic  in  our  natural  language  is  the 
most  important  and  pre-eminent  need.  Next,  a  recogni¬ 
tion  that  most  decisions  are  fundamentally  subjective 
since  they  are  imbedded  in  a  subjective  set  of  reference 
values ...  .  [H-6] 
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Exhibit  9 


Question  No.  10  (Q10) — Numerical  Summary 
The  Question's  Phrasing: 

There  is  no  use  for  a  decision-making  model  to  support 
top  level  decision-makers  because  they  base  their 
judgement  mainly  on  a  subjective  experience,  perception 
and  intuition  which  cannot  be  rationally  "managed". 


Table  A-V-3 

American  Responses  Distribution 


Distribution  (Amount) 


Low  Hi 


Distribution  (Percent 


Decidedly  Agree 
Moderately  Agree 
Perhaps  Agree 
Moderately  Disagree 
Decidedly  Disa 


Lew 

- 

4.3 

2.7 

35.7 

43.5 

40.6 

- 

8.7 

5.4 

35.7 

26.1 

29.7 

28.6 

17.4 

21.6 

The  answers  to  QlO  are  really  split:  43%  agree  with  the  state¬ 
ment  and  51%  disagree.  It  is  interesting  to  note  that  the 
person  at  the  highest  position  responding  to  the  question¬ 
naire  is  on  the  "disagree"  side,  i.e.,  thinks  that  such  models 
do  suit  top-level  decision-makers. 

In  the  Israeli  community  the  picture  is  completely  dif¬ 
ferent,  and  consistent  with  trends  observed  before.  91%, 
including  all  four  generals,  disagree  with  the  statement, 
i.e.,  think  that  quantitative  models  are  compatible  with  top 
level  decision-making. 

In  fact,  QlO  is  the  key  question  of  the  whole  questionnaire 
and  the  thesis  itself.  No  wonder  that  many  of  the  respondents 
added  written  comments  with  respect  to  this  issue.  Apparently, 
the  comments  reflect  various  insights,  which  emphasize  the 
fact  that  the  issue  is  much  more  complex  than  covered  by  the 
questions.  Here  are  some  citations  from  the  verbal  comments: 

One  respondent  finds  the  models  uncapable  of  handling 

judgment  and  intuition.  Moreover,  it  can't  handle  some  further 

factors  that  rule  the  decision-making  process: 

The  model  would  be  totally  perverted.  It  is  not  an  issue 
of  models — it's  a  matter  of  power  and  emotion  1  [H-3] 

Other  respondents  who  would  like  to  see  a  cross-section 

model  are  backed  down  by  the  model's  limitations  to  handle 

unlike  alternatives  (e.g.,  TOW  missile  vs.  A-10  a/c  to  kill 

tanks  or  fighter  a/c  vs.  SAM  for  air-defense  [H-4,L-1]). 

Another  respondent  emphasizes: 

The  personnel  experience  and  preference  of  senior 
officials  in  key  positions  plays  heavily  in  the 
decision  process .. .The  skill  of  the  program  sponsor 
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to  communicate  the  value  of  his  program  has  great  impact 
on  the  decision.  [H-7] 

Many  warn  that  the  model  might  be  structured  to  support 
a  "prior"  selected  alternative  [for  example  H-8,  L-5] ,  or  as 
put  in  more  biting  words: 

Use  of  such  models  is  phony  as  a  3  dollar  bill.  It's 
a  travesty  on  honest  quantification.  Anyone  who  would 
use  such  a  model  deserves  what  he  gets!  [H—  3 ] 

Many  view  models  as  a  useful  tool  for  decision-making 
[H-9,L-3,L-l,H-l,H-4  and  more],  but  with  an  emphasis  on  the 
word  "tool",  i.e.,  the  model  is  not  the  exclusive  determinant 
in  the  decision  process,  and  its  outcomes  do  not  dictate  the 
decision.  When  used,  its  weakness  should  be  recognized,  and 
taken  into  account. 

C.  PREFERRED  QUANTITATIVE  TECHNIQUE  TO  ASSIST  ACQUISITION 

DECISION  MAKING 

Q4  (see  Exhibit  10)  was  posed  to  explore  what  quantitative 
approach  or  technique  was  preferred  by  the  respondents.  The 
question  is  phrased  in  a  general  sense  and  does  not  address 
specific  decision  events,  procedures  or  level  of  decision. 

The  response  is  somewteit  surprising,  with  an  apparent  "one 
sided"  distribution  of  50%  favoring  the  MAUT  approach  (defined 
as  "scores  and  weights  assigned  to  factors") ,  while  none  of 
the  other  approaches  exceeds  15%  (!).  Very  similar  situation 
is  revealed  in  the  Israeli  community,  were  55%  favor  the  MAUT. 

Another  finding  is  that  23%  of  the  high-level  American 
respondents  have  never  used  any  quantitative  model  in  their 
decision-making.  4  respondents,  3  of  the  low-level  and  1  of 
the  high-level  categories,  chose  to  ignore  this  question. 
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Question  No.  4  (Q4) — Numerical  Summary 
The  Question's  Phrasing: 

If  you  ever  used  a  model  for  overall  evaluation  of 
acquisition  alternatives  (not  necessarily  at  mile¬ 
stones  decisions) ,  which  one  did  you  find  the  most 
useful? 

Table  A-V-5 

American  Responses  Distribution 


Distribution  (Amount 


Lew  Hi 


Distr 


Scores  &  Weights 
Decision  Tree 
Computerized 
Another 
Never  Used 


Did  not  respond  3  + 

Never  used  +  did  not  respond — 10  out  of  38 — 26.3%) 


Lew 

■SSnESESl 

75.0 

36.4 

50.0 

™  | 

18.2  | 

11.8 

16.7 

13.6  ! 

14.7 

- 

9.1 

5.9 

8.3 

22.7 

17.6 

n 


Scores  Dec .  Catpu-  Another  Never 

&  Weights  Tree  terized  Used 

Figure  A-V-3.  American  Responses  Distribution  Histogram 

Table  A-V-6 

Israeli  Responses  Distribution 


1  Distribution  (Amount)  Distribution  (Percent) 


6  |  54.5 


L  3  5 


r 


Since  the  question  offered  an  option  to  write  down  any  sort 
of  a  model,  one  can  think  that  the  ignoring  respondents,  in 
fact,  belong  to  the  "never  used"  category.  If  so — percentages 
are  obviously  higher  (26%) . 

D.  THE  COMPATIBILITY  OF  MAUT  TO  TOP-LEVEL  ACQUISITION 

DECISION-MAKING 

Q17  (see  Exhibit  11) ,  phrased  in  a  negative  way,  states 
that  "such  a  model  [MAUT  model,  R.G.]  is  not  useful  for  top- 
level  decision-making."  The  answers  are  clearly  "one-sided" 
distributed.  64%  are  on  the  "disagree"  side  while  only  14% 
are  on  the  opposite  side.  22%  are  in  the  middle.  This  obvious 
favorable  attitude  towards  the  MAUT  usage  for  top-level  decision¬ 
making  raises  some  wonders  as  compared  to  QlO's  "split"  dis¬ 
tribution,  in  which  44%  expressed  their  reservation  towards 
such  usage.  To  examine  this  phenomenon  a  test  for  consistency 
took  place.  The  results  are  clear:  of  those  who  were  in 
favor  of  model  usage  by  top-levels  per  Q10,  70%  remained  con¬ 
sistent  in  their  answer  to  Q17,  18%  changed  to  a  middle  choice, 
and  only  12%  completely  changed  their  minds.  Of  those  who 
thought  models  are  not  compatible  for  top-levels  in  Q10,  only 
19%  remained  consistent  in  Q17,  while  56%  (!)  completely  changed 
their  minds  and  25%  moved  to  the  middle  choice.  What  caused 
this  inconsistency  is  hard  to  tell.  Perhaps  the  corresponding 
question’s  wording  caused  different  perceptions  than  those 
intended  by  the  writer  of  the  questions.  But  the  well  defined 
group  that  shifted,  while  all  other  respondents  are  stable,  re¬ 
duces  the  probability  that  this  is  the  case.  Those  who  thought 


186 


Exhibit  11 


Question  No.  17  (Q17) — Numerical  Summary 
The  Question's  Phrasing: 

Such  a  model  is  not  useful  for  top-level  decision¬ 
making  . 


Table  A-V-7 

American  Responses  Distribution 


Distribution 

(Amount) 

Distribution 

(Percent) 

Low 

maem 

Total 

Decidedly  Agree 

- 

3 

- 

13.0 

Moderately  Agree 

- 

2 

- 

8.7 

Perhaps  Agree 

4 

8 

30.8 

17.4 

22.2 

Moderately  Disagree 

5 

’  8 

13 

38.5 

34.8 

36.1 

Decidedly  Disagree 

4 

6 

10 

30.8 

26.1 

27.8 

Figure  A-V-4.  American  Responses  Distribution  Histogram 


Comment:  For  the  Israeli  response  see  No.  1  in  Table  A-V-12 
at  the  end  of  this  chapter. 


that  quantitative  models  might  be  useful  for  top  level  in  the 
first  place,  remained  firm  in  their  minds.  Those  who  opposed 
that  assumption — to  a  great  extent  changed  their  minds.  It 
may  be  that  the  actual  example  presented  in  the  questionnaire 
in  Part  IV  (after  Q10)  caused  the  shift  of  attitudes,  as  hinted 
by  one  of  the  comments:  "I  think  you  are  onto  something  in 
Part  IV"  [H-l] .  On  the  Israeli  side  the  consistency  is  much 
higher.  As  mentioned  before,  only  one  Israeli  respondent  agreed 
with  the  statement  of  Q10  while  an  overwhelming  majority  of 
91%  disagreed.  The  same  person  was  the  only  one  to  agree  with 
the  statement  of  Q17. 

Q18  (see  Exhibit  12)  corresponds  to  Q17,  except  that  it 
narrows  the  scope  to  SecDef  only.  Though  one  could  expect  that 
people  would  be  very  doubtful  about  the  usefulness  of  the  model , 
at  least  at  the  SecDef  level,  this  is  not  the  case.  The  general 
distribution  is  pretty  similar  to  that  of  Q 17  with  68%  disagree 
(i.e.,  do  think  that  the  model  is  useful  also  at  the  SecDef 
level,  although  he  deals  by  and  large  With  the  political  as¬ 
pects  of  the  decision),  23%  "perhaps  agree",  and  only  9%  agree. 
The  extent  of  disagreement  with  the  statement  is  even  greater 
than  in  Q17  since  here  37%  are  at  the  extreme  answer  of  "decid¬ 
edly  disagree"  vs.  28%  in  £17  and  no  one  selected  the  opposite 
extreme  vs.  8%  in  Q17.  Q27  (see  Exhibit  13)  examines  the  value 

of  the  MAUT  approach  for  any  case  of  multicriteria  decision. 

The  responses  are  distributed  according  to  the  "middle"  distri¬ 
bution  type.  The  greatest  portion  of  39%  prefer  the  "perhaps 
agree"  answer.  On  the  other  hand,  out  of  the  rest,  the  number 
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Question  No.  18  (Q18) — Numerical  Summary 
The  Question's  Phrasing: 

Such  a  model  is  useful  for  all  levels  except  SecDef, 
since  political  aspects  can't  be  handled  by  the 
model . 


Table  A-V-8 

American  Responses  Distribution 


— - 

' 

Distribution 

(Amount) 

Distribution 

(Percent)  i 

Lew 

Hiqh 

Total 

Lew 

High 

Total  ! 

Decidedly  Agree 

- 

- 

- 

- 

- 

! 

Moderately  Agree 

- 

3 

3 

13.6 

8.6 

Perhaps  Agree 

5 

3 

8 

38.5 

13.6 

22.9 

Moderately  Disagree 

3 

8 

11 

23.0 

36.4 

31.4 

Decidedly  Disagree 

5 

_ § _ 

A3 

38.5 

36.4 

37.1 

Figure  A-V-5.  American  Responses  Distribution  Histogram 


Comment:  For  the  Israeli  response  see  No.  2  in  Table  A-V-12 

at  the  end  of  this  chapter. 
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Question  No.  27  (Q2 7) --Numerical  Summary 
The  Question's  Phrasing: 

Such  a  model  is  a  very  good  way  to  reach  a  decision 
in  a  multicriteria  case. 


Table  A-V-9 

American  Responses  Distribution 


Distribution 

(Amount) 

Distribution 

(Percent) j 

low 

High 

Total 

-  Low 

j  High 

Total  1 

Decidedly  Agree 

1 

3 

I  7’8 

l  8.7 

8.3  | 

Moderately  Agree 

6 

13 

j  46.1 

30.4 

36.1 

i  Perhaps  Agree 

6 

14 

46.1 

1  34.8 

38.9 

!  Moderately  Disagree 

- 

4 

|  17.4 

11.1 

!  Decidedly  Disagree 

~ 

2 

_ 

i  8.7 

5.6 

Figure  A-V-6,  American  Responses  Distribution  Histogram 


Comment:  For  the  Israeli  response  see  No.  3  in  Table  A-V-12 

at  the  end  of  this  chapter. 


of  those  who  agree  with  the  statement  that  "such  a  model  is  a 
very  good  way  to  reach  a  decision  in  a  multicriteria  case",  is 
much  greater  than  that  of  those  who  disagree--44%  vs.  17% 
respectively.  Perhaps  the  large  amount  of  reservation  was 
caused  by  the  phrase  "very  good"  in  the  statement,  since  people 
usually  tend  not  to  commit  themselves  to  superlative  phrases. 
Several  respondents  commented  that  they  would  have  selected  a 
higher  degree  of  agreement  answer  if  the  word  "very"  would  have 
been  eliminated  [for  example  H-7,H-1Q]. 

Q15  and  Q16  (see  Exhioits  14  and  15  respectively)  in  fact 
state  the  same  idea  in  opposite  approaches,  negative  and  posi¬ 
tive.  The  responses  are  very  consistent,  and  for  both  questions 
the  distribution  is  of  the  "one  sided"  type.  In  both  questions 
the  respondents  express  their  belief  that  MAUT  is  useful  for 
acquisition  related  decisions.  The  rejection  of  the  negative 
statement  of  Q15  saying  that  "such  a  model  is  not  for  acquisi¬ 
tion  related  decisions",  is  very  clear — 75%  disagree  and  only 
9%  agree.  16%  perhaps  agree.  The  answers  to  the  positive 
statement  are  a  little  less  clear  cut  distribution:  50%  agree 
and  25%  disagree.  25%  are  in  the  middle.  Again  it  may  be 
that  the  usage  of  the  superlative  phrasing  'useful  in  most 
cases"  evoked  the  respondents  reluctance  to  select  clearly 
favorable  answers.  In  any  case,  the  favorable  attitude  toward 
MAUT  in  assisting  acquisition  decision-making  is  clearly  spelled 
out  by  the  responses  to  these  two  questions. 


191 


Exhibit  14 

Question  No.  15  (Q15) — Numerical  Summary 
The  Question's  Phrasing: 

Such  a  model  is  not  for  acquisition  related  decisions. 


Table  A-V-10 

American  Responses  Distribution 


— 

Distribution 

(Amount) 

Distribution  (Percent) 

Ipw  _ 

Hiqh 

Total 

low 

High 

Total 

Decidedly  Agree 

1 

i 

- 

4.3 

2.8 

Moderately  Agree 

1 

1 

2 

7.7 

4.3 

5.6 

Perhaps  Agree 

1 

5 

6 

7.7 

21.7 

16.7 

Moderately  Disagree 

6 

9 

15 

46.1 

39.1 

41.7 

Decidedly  Disagree 

5 

7 

12 

38.5 

30.4 

33.3 

Figure  A-V-7.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  responses  see  No.  4  in  Table  A-V-12 

at  the  end  of  this  chapter. 
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E.  CHAPTER  SUMMARY 

From  the  exploration  of  the  respondents  attitudes  toward 
quantitative  technique  use  in  assisting  top-level  decision¬ 
making  in  general,  and  acquisition  related  decisions,  in 
particular,  several  dominant  trends  are  identified.  Naturally, 
these  trends  do  not  represent  consensus,  but  in  most  cases  in 
this  chapter  they  are  clearly  deduced  from  the  answers.  These 
are: 

(1)  Quantitative  techniques  might  help  in  cases  of  diffi¬ 
cult  selection  between  alternatives. 

(2)  The  opinions  are  split  with  respect  to  the  general  use 
of  quantitative  techniques  by  high  level  decision-makers. 

(3)  The  MAUT  (Multi-Attribute  Utility  Theory)  is  highly 
preferred  by  the  responding  sample  over  any  other  quantitative 
technique,  for  acquisition  decisions. 

(4)  Dealing  with  the  specific  technique  of  Multi-Attribute 
Utility,  much  of  the  reservation  expressed  tr wards  general  use 
of  quantitative  models  disappears  and  the  respondents  are 
clearly  favorable  in  their  attitude: 

(a)  They  think  it  does  fit  top-level  decision-making, 
including  at  the  '  ">ef  level . 

(b)  They  tnx  -  it  is  useful  for  acquisition  related 
decisions . 

Thus,  the  respondents  support  the  questionnaire's  prior  assump¬ 
tion  that  MAUT  is  the  preferred  approach  among  quantitative 
techniques  that  might  assist  top-level  acquisition  decision¬ 
making.  [It  is  not  to  say  that  other  approaches,  some  of  them 
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compute *;i zed  (as  a  complex  MAUT  can  be  as  well)  ,  should  not 
be  used  at  the  various  phases  and  levels  of  evaluation  of 
decision  alternatives.]  Thus,  the  prior  emphasis  put  on 
exploration  of  the  MAUT  features  and  implementation  techniques 
in  the  questionnaire  may  be  considered  posteriorily  as  an 
unbiased  reflection  of  the  respondents'  preference. 


Table  A-V-12 


Statements  Selection  By  the  Israeli  Respondents  (Selection 
is  identical  to  agreement  with  the  statement.) 


NO. 

Corresponds 
to  "American" 
Question  # 

The  Statement 

Selected 

by 

(amount) 

Selected 

by 

(percent) 

1 

Q17 

Such  a  model  is  not  useful 
for  top  level  decision-naking 

1 

9 

2 

Q18 

Such  a  model  is  useful  for 
all  levels  except  Min.  of 
Defense  since  political  aspects 
are  not  eligible  for  quanti¬ 
fication 

2 

18 

3 

Q27 

In  spite  of  its  weaknesses 
such  a  model  is  the  best 
way  to  reach  a  decision  in 
a  multicriteria  case 

4 

36 

4 

Q15 

Such  a  model  is  not  for 
acquisition  related 
decisions 

1 

9 

5 

Q16 

Such  a  model  nay  be  useful 
in  most  cases  of  acquisition 
selection  decisions 

6 

55 
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A-VI .  Advantages  and  Disadvantages  of  MAUT 

A.  PURPOSE 

The  purpose  of  this  chapter  is  to  explore  the  advantages 
and  disadvantages  of  the  MAUT,  as  they  are  conceived  by  the 
questionnaire  respondents. 

While  the  essential  features  of  the  NAUT  capability  in 
handling  complex  multicriteria  decisions  has  been  explored  in 
the  previous  chapter,  this  chapter  deals  mostly  with  side- 
effects  of  this  technique. 

B.  CHARACTERISTICS,  ADVANTAGES  AND  DISADVANTAGES 

Q19  (see  Exhibit  16)  addresses  the  MAUT's  substantial  ap¬ 
proach  to  base  the  total  evaluation  on  aggregated  independent 
evaluations  done  through  separate  value-dimensions.  An  assump¬ 
tion  is  that  people  might  consider  such  an  approach  as  improper, 
preferring  more  sophisticated  examination  of  interrelations 
between  factors.  The  answers  distribution  to  the  question  is 
pretty  much  "one-sided":  53%  disagree  with  the  statement 
and  only  21%  agree.  26%  perhaps  agree.  Thus,  the  respondents 
do  think  that  evaluation  by  separate  factors  is  proper,  although 
by  that  answer  they  do  not  express  any  rejection  of  further 
inter- factor  evaluations,  using  other  methods. 

Q25  (see  Exhibit  17)  puts  light  on  one  of  the  prominent 
features  assumed  for  MAUT,  of  creating  a  common  denominator 
and  common  language  of  communication  between  the  people  in¬ 
volved  in  the  decision.  In  cases  where  "the  personal  experi¬ 
ence  and  preference  of  senior  officials  plays  heavily  in  the 
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Exhibit  16 


Question  No.  19  (Q19) — Numerical  Summary 
The  Question's  Phrasing: 

Evaluation  by  separate  factors-  is  not  appropriate, 
since  the  appropriate  evaluation  is  derived  from  the 
complex  interrelations  among  factors. 


Exhibit  17 

Question  No.  25  (Q25) — Numerical  Summary 

The  Question's  Phrasing: 

Such  a  model  enables  better  communication  among 
people  involved  in  the  decision-making. 

Table  A-VI-2 

American  Responses  Distribution 


Decidedly  Agree 
Moderately  Agree 
Perhaps  Agree 
Moderately  Disagree 
Decidedly  Di 


■S53H 

7 

30.8 

16 

38.5 

10 

30.8 

1 

- 

2 

- 

tion  (Percent) 


decision  process,  and  the  skill  of  the  program  sponsors  to 
communicate  the  value  of  the  program  has  a  great  impact"  [H-7] , 
MAUT  can  compensate  the  differences  in  experience  and  reduce 
the  impact  of  personal  communication  skills,  by  "offering  an 
easily  comprehended  standard  of  comparison"  [H-ll].  What  is 
deduced  from  the  above  citations  is  clearly  spelled  out  in  the 
answer  distribution,  where  64%  think  the  MAUT  model  enables 
better  communication,  and  only  8%  object. 

Still  "one-sided",  although  less  definit,  is  the  attitude 
toward  the  assumption  stated  in  Q21  (see  Exhibit  18)  saying 
that  "such  a  model  leads  people  to  refer  to  the  really  impor¬ 
tant  issues  in  the  decision."  50%  agree  with  the  statement, 

19%  disagree  and  a  relatively  high  percentage  are  inbetween, 
accounting  for  31%. 

The  respondents'  written  comments  in  this  respect  reveal 
the  controversy  about  this  issue.  On  the  one  hand  there  are 
comments  such  as : 

Those  who  have  used  such  a  model  often  find  that  it 
helps  them  come  to  grips  with  the  critical  factors  in 
selection.  [L— 6 ] 

Or, 

Basically,  a  model  is  useful  to  force  the  decision¬ 
maker  to  consider  and  weigh  all  the  factors"  [H-ll] , 

and 

...including  all  relevant  decision  factors  is  the 
tough  part.  [H-l] 

Or, 

Models  similar  to  the  one  you  described  are  helpful 
in  focusing  attention  on  critical  issues  in  a  dis¬ 
ciplined  way.  [H-12] 
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Exhibit  18 


Question  No.  21  (Q21) — Numerical  Summary 
The  Question  Phrasing: 

Such  a  model  leads  people  to  refer  to  the  really 
important  issues  in  the  decision. 


Table  A-VI-3 

American  Responses  Distribution 


Distribution 

(Amount) 

Distribution 

(Percent) 

Total 

L CM 

High 

j  Total  : 

Decidedly  Agree 

3 

4 

7 

23.1 

17.4 

i  19-4 

Moderately  Agree 

3 

8 

11 

23.1 

34.8 

j  30.6 

Perhaps  Agree 

6 

5 

11 

46.2 

21.7 

'  30.6 

Moderately  Disagree 

1 

4 

S 

7.7 

17.4 

j  13.9 

1  Decidedly  Disagree 

- 

2 

2 

- - 

3.7 

!  5.6 

Figure  A-VI-3.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  3  in  Table  A-VI-10 

at  the  end  of  this  chapter. 
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But  on  the  other  hand,  there  are  those  who  think  differently 
as  reflected  from  the  following  comments: 


Models  tend  to  amplify  irrelevant  data  with  respect 
to  relevant  data.  [H-6] 

Or, 

Model  can  serve  to  focus  debate,  but  it  could  mask 
items  not  addressed. 

There  are  commentators  who  recognize  MAUT ' s  capability  to 

throw  light  on  the  important  issues,  but  warn  against  a 

tendency  to  rely  totally  on  the  models  results  in  making 

the  decision.  Here  are  some  examples: 

Such  a  model  would  give  useful  insight  into  the  choice 
being  made,  but  would  never  be  the  circumference  of 
insight.  [H— 1 3 ] 

Or, 

Model  is  simply  a  tool  to  summarize  results  of 
evaluation.  [H— 14 ] 

The  conclusion  drawn  from  this  variety  of  responses  might  be 
that  MAUT  has  the  potential  to  focus  attention  on  the  impor¬ 
tant  issue  and  to  provide  insight  into  alternatives  if 
properly  used.  Being  aware  of  potential  pitfalls  like  includ¬ 
ing  irrelevant  data  or  excluding  significant  items,  one  can 
exploit  the  technique's  advantages.  All  this  is  true  unless 
the  model  is  biased  or  deliberately  structured  to  support 
priorly  selected  alternatives.  As  put  by  one  of  the  respondents 

Model  would  be  very  useful  unless  the  decision  is 
proformula.  [H-2] 

Th^  question  to  what  extent  MAUT  can  reduce  the  effect  of  a 
biased  presentation  of  alternatives  is  addressed  in  the  body 
of  the  thesis. 
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Another  tentative  feature  of  MAUT  is  in  helping  people 
to  manage  their  thinking  and  judgment  in  a  logically  organized 
manner,  which  naturally  helps  the  decision-making.  This  feature 
is  addressed  by  Q20  (see  Exhibit  19).  Again,  the  favorable 
attitude  is  clearly  spelled  out  from  the  answers  distribution: 

67%  agree,  only  8%  disagree,  25%  perhaps  agree.  Here  the 
notion  of  judgment  is  emphasized.  The  question  is  to  what 
extent  a  technique  which  is  designed  to  handle  judgment  ("what 
appears  to  be  'quantitative'  is  in  reality  subjective  as  both 
the  weights  and  the  rating  are  based  on  judgment"  [L-7]) , 
appeals  to  the  decision-makers  who  based  their  decisions  by 
and  large  on  judgment  ("In  my  experience — most,  if  not  all 
acquisition  decisions  are  a  matter  of  judgment"  [H-l]).  It 
seems  that  the  respondents  had  no  doubts  about  the  answer  to 
that  question. 

Q29  (see  Exhibit  20)  addresses  a  controversial  issue  as 
observed  from  the  answers'  distribution.  The  question  indi¬ 
cates  the  possibility  that  the  discussion  would  concentrate 
on  figures  of  weights  and  scores  (also  called  "utility  measures", 
or  "location  measures") ,  which  one  may  consider  as  diversion 
from  the  essential  issues  under  the  decision.  The  respondents 
split  on  this  issue  very  sharply:  46%  agree,  40%  disagree 
and  only  14%  are  in  the  middle. 

The  written  comments  express  the  respondents'  reservation: 

While  the  use  of  a  model  may  help  to  more  fully  evalu¬ 
ate  the  differences  between  alternatives,  I  feel  that 
your  observation  that  a  model  may  shift  emphasis  from 
significant  issues  to  a  mere  discussion  of  weights  and 
scores  is  the  critical  flaw.  [L-8] 


203 


Exhibit  19 


Question  No.  20  (Q20) — Numerical  Summary 
The  Question's  Phrasing: 

The  model  leads  people  to  manage  their  judgment  in  a 
logical  manner,  which  helps  the  decision-making. 


Table  A- VI-4 

American  Responses  Distribution 


I 

I 

Distribution  i 

(Amount)  i 

Distribution 

(Percent)  i 

Low 

Total 

j  Decidedly  Agree 

3 

6 

9 

^  i 

23.1 

26.1 

25.0  ; 

|  Moderately  Agree 

4 

11 

15 

30.1 

47.8 

41.7  , 

|  Perhaps  Agree 

5 

4 

9 

38.5 

j  17.4 

!  25.0  , 

j  Moderately  Disagree 

1 

1 

7.7 

1  ^ 

i  2-8  , 

1  Decidedly  Disagree 

- 

2 

! _ £ _ 1 

2 

! _ I _ 

!  8.7 

5.6 

Figure  A-VI-4 .  American  Responses  Distribution  Histogram 


Comment:  For  the  Israeli  response  see  No.  4  in  Table  A-VI-10 
at  the  end  of  this  chapter. 
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It  seems  that  the  controversy  is  mostly  on  the  word  "unnecessary", 
since  arguments  on  scores  and  weights  and  the  rationale  behind 
them  can  be  arguments  on  the  very  essence  of  the  decision. 

So  the  question  is  about  the  possibility  that  they  would  be¬ 
come  "unnecessary".  Just  awareness  can  probably  reduce  such 
occurrence. 

Q22,  A23  and  Q24  {see  Exhibits  21,  22,  23,  respectively) 
focus  on  the  notion  that  structuring  the  evaluation  into  a 
model  (or  a  procedure)  framework,  and  presenting  it  as  such 
to  the  decision-maker,  may  suppress  his  natural  intuition, 
bias  his  judgment,  and  bind  his  decision-making  freedom.  The 
respondents  explicitly  reject  these  three  potential  deficien¬ 
cies.  The  question  statements  are  all  positively  stated,  thus 
disagreement  means  rejection.  The  respective  distributions 
for  the  three  questions  are:  72%  disagree,  20%  agree,  8%  per¬ 
haps  agree;  80%  disagree,  12%  agree,  8%  perhaps  agree;  63% 
disagree,  23%  agree,  14%  perhaps  agree.  One  can  notice  that 
the  third  distribution  is  a  little  less  "one-sided"  relative 
to  the  other  two.  This  is  caused,  perhaps,  by  the  implications 
of  the  use  of  decision  tools  and  not  because  of  the  specific 
nature  of  the  MAUT  procedure.  As  mentioned  by  one  of  the 
respondents : 

If  a  decision-maker  embraces  a  model--but  then  dis¬ 
agrees  with  the  model  outcome,  he  may  be  baring  his 
belly  to  his  foes.  [L-5] 

This  feature  is  addressed  in  Q26  (see  Exhibit  24)  where  the 
emphasis  is  put  on  the  decision  criticism,  using  the  model 
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Exhibit  21 


Question  No.  22  (Q22) — Numerical  Summary 

The  Question's  Phrasing: 

Such  a  model  suppresses  the  natural  intuitive 
judgment. 


Table  A-VI-6 

American  Responses  Distribution 


Distribution  (Amount) 

Distribution 

(Percent)  i 

Low 

WEEM 

■sre™ 

m&jm 

Total  I 

Decidedly  Agree 

1 

1 

4.3 

2.8  | 

Moderately  Agree 

2 

4 

6 

15.4 

17.4 

16.7 

Perhaps  Agree 

1 

2 

3 

7.7 

8.7 

8.3  I 

Moderately  Disagree 

6 

11 

17 

46.2 

47.8 

47.2 

Decidedly  Disagree 

4 

5 

>_J _ 

30.8 

21.7 

25.0 

Figure  A-VI-6.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  6  in  Table  A-VI-10 

at  the  end  of  this  chapter. 


Exhibit  22 


Question  No.  23  (Q23) — Numerical  Summary 

The  Question's  Phrasing: 

Such  a  model  may  bias  the  decision-maker's  good 
j  udgment . 


Table  A-Vi-7 

American  Responses  Distribution 


Low 

to* 

Total  i 

Decidedly  Agree 

- 

2 

2 

9.1 

5.7  j 

Moderately  Agree 

1 

1 

2 

7.7 

4.6 

5.7 

I 

Perhaps  Agree 

2 

1 

3 

15.4 

4.6 

8.6  | 

Moderately  Disagree 

9 

14 

23 

69.2 

63.6 

65.7  ; 

Decidedly  Disagree 

L-.l— 

4 

5 

7.7 

18.2 

14.3 

Figure  A-VI-7.  American  Responses  Distribution  Histogram 


■  Comment:  for  the  Israeli  response  see  No.  6  in  Table  A-VI-10 

j  at  the  end  of  this  chapter. 

i 
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Exhibit  23 


Question  No.  24  (Q24) — Numerical  Summary 
The  Question's  Phrasing: 

Such  a  model  may  somehow  bind  the  decision-maker's 
freedom. 


Table  A-VI-8 

American  Responses  Distribution 


Distribution 

(Amount) 

ni 

Lew 

ws&m 

lew 

High 

Total  I 

Decidedly  Agree 

- 

i 

1 

4.6 

2.9  ! 

Moderately  Agree 

3 

4 

7 

18.2 

20.0  i 

Perhaps  Agree 

- 

5 

5 

22.8 

14.3 

Moderately  Disagree 

8 

8 

16 

36.4 

45.7  ; 

Decidedly  Disagree 

2 

4 

6 

18.2 

17.1 

Figure  A-VI-8.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  7  in  Table  A-VI-10 
at  the  end  of  this  chapter. 


Exhibit  24 


Question  No.  26  (Q26) — Numerical  Summary 
The  Question's  Phrasing: 

Such  a  model  may  be  used  as  a  tool  to  criticize  the 
decision  (e.g.,  by  Congress),  and  therefore 
undesirable. 


Table  A- VI-9 

American  Responses  Distribution 


I 3SS 

Distribution 

[Percent) 

Low 

■[2SS9I 

Low 

High 

Total 

Decidedly  Agree 

- 

i 

i  1 

4.6 

2.9 

Moderately  Agree 

3 

7 

10 

31.8 

29.4 

Perhaps  Agree 

2 

2 

4 

9.1 

11.8 

Moderately  Disagree 

5 

7 

12 

31.8 

35.3 

Decidedly  Disagree 

2 

5 

7 

22.7 

20.6 

j 


Figure  A-VI-9.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  8  in  Table  A-VI-10 

at  the  end  of  this  chapter. 


outcomes,  by  bodies  external  to  the  decision-making  community. 
In  fact,  it  can  be  used  by  "in-house"  rivals,  as  well. 

The  answers  to  Q26  are  split.  Although  the  majority  of 
56%  do  not  think  that  such  a  "danger"  makes  the  model  unde¬ 
sirable,  a  significant  portion  of  32%  do  think  so.  One  of  the 
respondents  suggests, 

...that  model  results — if  publically  available — could 
compromise  the  choice  when  the  debate  starts... the 
quantitative  results  should  be  privately  held.  [H-ll] 

But  there  is  also  the  one  who  thinks  that  the  "external" 

control  on  decisions  acts  as  an  incentive  for  models  use: 

. . .With  the  Congressional  oversight  of  the  decision¬ 
making  process,  backup  of  decision  [by  a  quantitative 
model,  R.G.]  is  virtually  mandatory.  [L-9] 

C.  CHAPTER  SUMMARY 

The  analysis  of  this  chapter's  responses  reveals  several 
favorable  side-effects  of  the  MAUT  procedure: 

(1)  It  allows  better  communication  and  understanding  among 
the  people  involved  in  the  decision-making. 

(2)  It  has  the  potential  to  focus  discussion  on  the  impor¬ 
tant  issues. 

(3)  It  assists  decision-makers  in  putting  order  into  their 
line  of  thinking. 

Some  important  cautions  are  raised  by  the  commentators: 

(1)  Not  to  include  irrelevant  data. 

(2)  Not  to  overlook  significant  attributes. 

(3)  Not  to  consider  the  model  as  the  whole  circumference 
of  the  decision,  rather  as  a  mere  tool. 
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The  respondents  expressed  their  confidence  that  a  quantitative 
technique  used  as  a  decision  tool  does  not  suppress  the 
decision-maker's  intuition,  bias  his  judgment,  or  bind  his 
freedom.  That  is  because  either  the  model  does  not  incline 
to  do  so,  or  because  decision-makers  are  immune  to  such 
tendencies,  or  both. 

'  .;*1  ly ,  the  respondents  are  aware  of  the  case  in  which 
.0  iel  results  may  be  used  as  a  tool  to  criticize  the 
:oj.on,  but  the  majority  do  not  think  that  this  should 
necessarily  cause  model  undesirability. 


Table  A- VI- 10 


Statements  Selection  3y  the  Israeli  Respondents  (Selection 
is  identical  to  agreement  with  the  statement) 


No. 

Corresponds 
to  "American" 
Question  # 

The  Statement 

Selected 

by 

(Amount) 

Selected 

by 

(Percent) 

1 

Q19 

Evaluation  based  on  separate 
independent  factor  evaluations 
is  inproper 

0 

0 

2 

Q25 

Such  a  model  enables  better 
ccmnunication  among  people 
involved  in  the  decision¬ 
making 

5 

45 

3 

Q21 

The  model  makes  clear  what 
is  really  important  to  the 
decision-making 

3 

27 

4 

Q20 

The  model  forces  the  people 
involved  to  orderly  arrange 
their  judgment  and  preferences 

6 

55 

5 

Q29 

Such  a  model  nay  shift  the 
discussion  fran  the  signifi¬ 
cant  issues  to  arguments  on 
weights  and  scores 

4 

36 

6 

Q22 

Such  a  model  suppresses  the 
natural-intuitive  judgment 

0 

0 

7 

Q24 

Such  a  model  nay  scmehow 
bind  the  decision-maker's 
freedom 

4 

36 

8 

Q26 

Such  a  model  may  be  used  as 
a  tool  to  criticize  the 
decision  by  external  bodies 

9 

82 
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A-VII.  Implementation  of  MAUT  Procedure 

A.  PURPOSE 

The  purpose  of  this  chapter  is  to  examine  how  the  ques¬ 
tionnaire  respondents  perceive  MAUT  procedure  implementation, 
in  its  use  as  top-level  acquisition  decision-making  tool. 

B.  ANALYSIS 

The  first  question  in  this  chapter,  Q28  (see  Exhibit  25) , 
refers  to  the  nature  of  the  model.  Having  in  mind  methods 
like  "Goeller  Scorecard"  or  the  "colors  method”  (using  red, 
green  and  yellow  to  denote  relative  values) ,  the  question 
states  that  a  score  for  each  factor  is  sufficient  and  the 
aggregation  of  utility  for  the  overall  preference  should  be 
left  for  a  pure  judgment.  The  response  to  this  statement  is 
clear  cut.  83%  disagree  and  only  6%  agree.  11%  are  in  the 
middle.  Only  1  Israeli  agreed  with  the  statement.  Thus,  the 
preference  is  clear:  if,  at  all,  to  use  such  a  procedure,  it 
should  go  all  the  way  through,  including  the  aggregate  utility 
derived  from  the  summation  of  the  products  of  weight  times 
the  associated  scores,  by  each  of  the  attributes. 

The  next  two  questions  deal  with  the  eligibility  to  change 
the  model's  components  and  by  that  to  affect  its  outcomes.  Q13 
(see  Exhibit  26)  puts  this  idea  in  the  negative  way  saying 
that  the  model  must  not  be  changed.  The  answer's  distribution 
is  almost  completely  "uniform"  with  equal  percentages  in  all 
five  degrees  of  agreement.  Q14,  (see  Exhibit  27),  whose  state¬ 
ment  "allows"  the  decision-maker  to  change  weights  and  score 
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Exhibit  25 


Question  No.  28  (Q28) — Numerical  Summary 
The  Question's  Phrasing: 

There  is  no  need  for  weights  and  total  score.  A 
score  for  each  factor  separately  is  sufficient. 
The  rest  should  be  left  for  pure  judgment. 


Table  A-VII-1 

American  Responses  Distribution 


■ 

m 

Distribution 

(Percent)  | 

Low 

Eaa 

Low 

Decidedly  Agree 

1 

1 

2 

7.7 

4.6 

5.7 

Moderately  Agree 

- 

- 

- 

- 

- 

- 

Perhaps  Agree 

2 

2 

4 

15.4 

9.1 

11.4 

Moderately  Disagree 

8 

13 

21 

61.6 

59.1 

60.0 

Decidedly  Disagree 

2 

6 

_ § _ 

15.4 

27.3 

22.9 

Figure  A-VII-1.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  1  in  Table  A-VII-10 

at  the  end  of  this  chapter. 
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Exhibit  26 

Question  No.  13  (Q13) — Numerical  Summary 
The  Question's  Phrasing: 

The  model  is  supposed  to  be  objective  therefore  may 
not  be  changed.  (The  decision-maker  is  not  bound, 
of  course,  by  the  model  outcomes.) 


Table  A-VII-2 

American  Responses  Distribution 


- - 

Distribution 

(Amount) 

Distribution 

(Percent)  , 

Low 

■2E m 

Low 

i  Hiqh 

Total 

Decidedly  Agree 

1 

5 

6 

7.7 

|  23.8 

17.6 

Moderately  Agree 

3 

3 

6 

23.1 

14.3 

17.6 

Perhaps  Agree 

2 

4 

6 

15.4 

19.0 

17.6 

Moderately  Disagree 

4 

5 

9 

30.8 

i  23.8 

26.5 

Decidedly  Disagree 

3 

_ 1_ 

7 

23.1 

i  19.0 

20  -l.  J 

Figure  A-VII-2.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  2  in  Table  A-VII-10 

at  the  end  of  this  chapter. 
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Exhibit  27 


Question  No.  14  (Q14) — Numerical  Summary 

The  Question's  Phrasing: 

The  model  is  prepared  by  an  analytical  process 
through  the  different  levels,  but  the  decision¬ 
maker  may  change  weights  and  scores  (interact  with 
the  model)  according  to  his  judgment. 


Table  A-VII-3 

American  Responses  Distribution 


. 

1 

Distribution 

(Amount) 

•Distribution 

[Percent)  | 

lew 

High 

Total 

Low 

High 

Total 

Decidedly  Agree 

7 

10 

17 

53.8 

47.6 

50.0 

Moderately  Agree 

3 

6 

9 

,  23.1 

28.6 

Perhaps  Agree 

2 

2 

“ 

9.5 

5.y  ; 

Moderately  Disagree 

3 

1 

4 

|  23.1 

4.8 

11  •  8 

Decidedly  Disagree 

- 

2 

2 

i 

9.5 

L  Jbi _ ; 

Figure  A-VII-3.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  3  in  Table  A-VII-10 

at  the  end  of  this  chapter. 


according  to  his  judgment  when  he  is  presented  a  structured 
model  breaks  this  tie.  Here  the  distribution  is  extremely 
"one-sided":  76%  agree  with  the  decision-maker's  freedom  to 

change  the  model.  Furthermore,  of  these  76%,  50%  decidedly 
agree,  an  unusual  amount  for  the  extreme  choice.  Another  issue 
is  how  the  decision-maker  should  actually  make  such  changes. 

No  doubt,  there  is  firm  basis  for  one  of  the  commentators 
observation: 

Usually,  I  do  not  think  the  decision-maker  will  openly 
or  overtly  change  weights  and  scores.  To  do  so  at 
the  'end'  only  invites  allegations  of  improper  rigging 
of  selection. 

Thus,  the  decision-maker's  interaction  with  the  model  requires 
delicate  and  wise  manipulation  of  the  decision  process  and 
the  decision  group. 

The  following  set  of  questions  deal  with  techniques  of 
determining  weights  and  scores  or  rather — who  should  deter¬ 
mine  them.  Four  alternatives  are  presented  by  the  questions: 

(1)  The  decision-maker  himself  should  determine  both 
weights  and  scores. 

(2)  The  decision-maker  should  determine  only  weights  (while 
scores  are  determined  otherwise,  presumably  at  lower  levels) . 

(3)  People  involved  (i.e.,  staff  members,  decision  group 
members  and  the  like)  should  determine  weights  and  scores. 

(4)  Group  of  experts  should  determine  weights  and  scores. 

In  fact,  these  alternatives  are  not  mutually  exclusive  and 
some  or  all  of  them  can  be  used  at  the  same  procedure  through 
its  various  phases  and  iterations.  Moreover,  there  are  many 
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more  combinations  of  "by  whom"  and  "how"  scores  and  weights 
are  determined  than  these  four  alternatives.  The  respondents 
had  to  cope  with  these  limitations,  which  resulted  in  a  par¬ 
tial  picture.  In  any  case,  the  respondents  were  aware  that 
the  four  alternatives  might  not  suffice.  58%  pointed  at  Q12 
(see  Exhibit  28)  to  the  decision-maker  as  the  one  who  should 
determine  weights  and  scores.  Only  25%  disagree,  and  17%  did 
not  have  a  definite  opinion.  About  the  same  distribution  oc¬ 
curred  in  Q31  (see  Exhibit  29),  suggesting  that  only  weights 
should  be  determined  by  the  decision-maker:  56%  agreed,  32% 
disagreed  and  12%  were  undecided  ("perhaps  agree").  Compared 
to  the  high  agreement  about  the  decision-maker  as  being  the 
weights  and  scores  determinator,  it  is  interesting  to  observe 
the  respondents  rejection  of  the  other  alternatives,  namely: 
the  determination  done  through  an  aggregation  of  individual 
evaluation  of  people  involved  in  the  decision  (Q33,  see  Exhibit 
30) ,  or  by  a  group  of  experts  (Q32,  see  Exhibit  31) .  In  the 
latter  case,  there  is  a  "middle"  distribution  with  the  largest 
group  of  32%  of  the  answers  at  the  "perhaps  agree"  row.  Differ 
ent  is  the  Israeli  attitude  toward  the  experts  alternative. 

64%  of  the  Israeli's  favor  this  alternative.  It  may  be  that 
the  apprehension  of  a  biased  model  underlies  the  American  reser 
vation  towards  any  kind  of  aggregate  opinions  in  structuring 
the  model.  But  perhaps  the  rejection  is  caused  just  by  unaware' 
ness  of  the  techniques  usually  implemented  in  MAUT.  And  it  may 
be  that  this  is  a  mere  result  of  poor  questioning  (caused  by 
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Exhibit  28 


Exhibit  29 


Question  No.  31  (Q31) — Numerical  Summary 
The  Question's  Phrasing: 

The  weights  are  the  main  ingredient  in  the  model 
outcome,  therefore  they  should  be  determined  by 
the  decision-maker  himself. 


Table  A-VII-5 

American  Responses  Distribution 


! 

(Amount) 

Low 

W52RM 

Lew 

ESDI 

Total  ) 

Decidedly  Agree 

1 

6 

7 

8.3 

27.3 

20.6 

Moderately  Agree 

5 

7 

12 

41.7 

31.8 

35.3 

Perhaps  Agree 

2 

2 

4 

16.7 

9.1 

11.8 

Moderately  Disagree 

4 

4 

8 

33.3 

18.2 

23.5 

Decidedly  Disagree 

3 

-  3  _ 

- 

13.6 

8.8 

Exhibit  30 

Question  No.  33  ( Q 3 3 ) — Numerical  Summary 
The  Question's  Phrasing: 

Weights  and  scores  should  be  determined  by  summing 
up  assignments  made  individually  by  people  involved 
in  the  decision. 


Table  A-VII-7 

American  Responses  Distribution 


Distribution 

(Amount) 

i  Distribution 

(Percent)  j 

Lew 

High 

!  Total 

Lew 

Total  j 

Decidedly  Agree 

3 

|  3 

14.3 

9.1  i 

Moderately  Agree 

3 

3 

6 

25.0 

14.3 

18.2  ; 

Perhaps  Agree 

2 

5 

j  1 

,  16.7 

23.8 

21.2 

Moderately  Disagree 

6 

5 

12 

50.0 

28.6 

36.4 

Decidedly  Disagree 

.1 

4 

i_§ _ 

:  8.3 

19.0 

15.2  | 

Figure  A-VII-7.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  resDonse  see  No.  5  in  Table  A-VII-10 

at  the  end  of  this  chanter. 
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Exhibit  31 


Question  No.  32  (Q32) — Numerical  Summary 
The  Question's  Phrasing: 

Weights  and  scores  should  be  determined  by  a  common 
work  of  a  group  of  experts. 


Table  A-VII-6 

American  Responses  Distribution 


— 

Distribution 

(Amount) 

Distribution  (Percent)  J 

Lew 

High 

Lew 

High 

j  Total 

Decidedly  Agree 

1 

2 

3 

8.3 

9.1 

8.8 

Moderately  Agree 

4 

i 

5 

33.3 

4.6 

14.7  | 

Perhaps  Agree 

3 

8 

11 

25.0 

36.4 

1  32.4  | 

Moderately  Disagree 

2 

6 

8 

16.7 

27.3 

j  23.5 

Decidedly  Disagree 

2 

5 

_ z _ 

16.7 

22.3 

1  20.6 

Figure  A-VII-6.  American  Responses  Distribution  Histogram 
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the  author's  unawareness  of  the  MAUT  techniaues  at  the  time 
of  the  composition  of  the  Questionnaire...)* 

A  "uniform"  distribution  is  observed  in  030  (see  Exhibit 
32) .  dealino  with  whether  or  not  weights  and  scores  should  be 
assigned  on  different  occasions  by  different  people.  The  dis¬ 
tribution  reflects,  perhaps,  the  variety  of  ways  to  maintain 
the  MAUT  procedure.  But  perhaps  it  is  again  the  lack  of 
experience  in  MAUT  implementation  that  causes  people  to  arbi¬ 
trarily  select  their  answer  all-over  the  spectrum. 

In  contrast  to  the  balanced  distribution  in  Q30,  the 
preference  is  very  clear  in  Qll  (see  Exhibit  33) — the  last 
one  in  the  implementation  section.  72%  of  the  respondents 
express  their  view  that  the  numerical  framework  of  the  MAUT 
procedure  is  not  sufficient  for  decision-making.  The  model 
presentation  before  the  decision-maker  should  be  supplemented 
by  the  underlying  data,  rationale  and  methodology  that  brought 
about  the  model's  weighting,  rating  and  overall  preference. 
Only  17%  disagree  with  the  above  idea,  and  11%  are  in  the 
middle . 

C.  ADDITIONAL  PRACTICAL  COMMENTS 

The  written  comments  made  by  the  questionnaire  respondents 

raise  some  additional  desirable  features  in  the  MAUT  procedure 

which  are  not  covered  by  the  questions.  The  following  quotes 

introduce  these  features : 

...A  range  of  subjective  values  and  numerical 
estimates  need  to  be  presented  to  top  decision¬ 
makers,  with  uncertainties  clearly  spelled  out. 

[H-13] 
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Exhibit  32 


Question  No.  30  (Q30) — Numerical  Summary 
The  Question's  Phrasing: 

Weights  and  scores  should  be  assigned  independently 
from  each  other  at  a  different  time,  and  perhaps, 
by  different  people. 


Table  A-VII-8 

American  Responses  Distribution 


1-1 

Distribution 

(Percent) 

Low 

High 

Low 

Total 

Decidedly  Agree 

3 

4 

7 

25.0 

18.2 

20.6 

Moderately  Agree 

4 

4 

8 

33.3 

18.2 

23.5 

Perhaps  Agree 

n 

5 

5 

- 

22.7 

14.7 

Moderately  Disagree 

n 

3 

7 

33.3 

13.6 

20.6 

Decidedly  Disagree 

HU 

6 

7 

8.3 

27.3 

20.6 

Figure  A-VII-8.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  6  in  Table  A-VII-10 

at  the  end  of  this  chapter. 
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Exhibit  33 


Question  No.  11  (Qll) — Numerical  Summary 
The  Question's  Phrasing: 

Such  a  model  may  be  used  only  if  supplemented  by 
representation  of  the  detailed  data  for  each  topic. 


Table  A-VII-9 

American  Responses  Distribution 


Distribution 

(Percent) 

Low 

■2231 

Lew 

Total 

Decidedly  Agree 

1 

— 

12 

19 

53.8 

52.2 

52.8 

Moderately  Agree 

4 

3 

7 

30.8 

13.0 

19.4 

Perhaps  Agree 

2 

2 

4 

15.4 

8.7 

11.1 

Moderately  Disagree 

4 

4 

17.4 

11.1 

Decidedly  Disagree 

L.  — 

2 

2 

8.7  , 

5.6 

100 


Figure  A-VII-9.  American  Responses  Distribution  Histogram 


Comment:  for  the  Israeli  response  see  No.  7  in  Table  A-VII-10 
at  the  end  of  this  chapter. 
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The  model  can  be  improved  by  specifying  at  the  outset 
that  the  scores  must  exceed  some  threshold  value  for 
selected  'critical*  factor.  [H-4] 

Go-No-Go  factors  should  be  identified  at  early 
stages,  especially  those  perceived  by  the  decision¬ 
maker,  which  usually  tend  to  be  revealed  only  at 
the  end.  [L-10] 

The  model  should  be  expanded  to  include  probabilities 
since  most  of  the  decisions  deal  with  the  future 
where  evaluations  are  mere  estimates .. .Also  constraints 
should  be  included  in  the  model.  [L-ll] 

Two  main  requirements  are  raised  by  the  above  comments: 

(1)  Uncertainties  and  risks  should  be  treated  by  the  model, 
and  clearly  presented  to  the  decision-maker. 

(2)  The  model  should  maintain  thresholds  (or  constraints) 

for  the  critical  factors.  Another  set  of  comments  relates  to 

the  "relationship"  between  the  model  and  the  decision-maker. 

The  following  quotes  might  illustrate: 

The  model,  i.e.,  the  structured  decision  process  is 
a  useful  tool,  but  it  must  be  acceptable  to  final 
decision-makers,  and  emphatically  not  plugged  into 
a  computer  or  be  defined  by  'experts' — specifically 
weights  and  scores.  [H-l] 

I  have  found  that  judgement  of  experienced  members 
of  the  decision-group  ("intuition")  is  a  dominant 
factor.  Analysis  and  models  are  steered  by  comments 
of  these  participants.  "Good"  decision-making  is 
enhanced  by  bringing  them  in  early,  so  that  data  base 
etc.,  is  organized  to  their  "taste".  [H-15] 

The  conclusion  is  that  the  decision-maker  should  be  consulted 

at  an  early  stage  of  the  evaluation  procedure  structuring  to 

ensure  his  acceptance  of  this  procedure.  Furthermore,  he  should 

affect  the  important  details  of  the  procedure  according  to  his 

experience,  intuition  and  preferences. 

Another  requirement  is  the  model  simplicity.  It  should 

be  easy  to  present  in  a  decision  session,  easy  to  grasp  by 
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decision-makers  whose  expertise  is  not  necessarily  in  the  very 
details  of  the  alternatives  presented  or  in  sophisticated 
modeling  procedures.  Models  that  "hide"  their  procedures 
within  a  computer  or  can  be  understood  only  by  the  experts 
who  built  them  have  little  value  for  top-level  decision-makers, 
if  any. 

D.  CHAPTER  SUMMARY 

After  a  reasonable  rounding  of  some  vague  answering  trends, 
the  following  conclusions  reflect  the  general  preferences  of 
the  respondents : 

(1)  The  model  is  subject  to  change  by  the  decision-maker. 

The  latter  should  implement  such  changes  in  a  way  that  would 
be  acceptable  as  a  proper  procedure. 

(2)  The  decision-maker  is  preferred  as  the  authority  to 
determine  weights  and  scores.  There  are  reservations  to 
parameters  determination  being  done  by  people  involved  in  the 
decision,  or  by  a  group  of  experts. 

(3)  A  presentation  of  the  MAUT  procedure  should  be  supple¬ 
mented  by  the  data,  rationale  and  methodology  which  under ly 
the  numerical  results. 

Some  additional  requirements  are  raised  through  the  verbal 
comments  to  the  questions : 

(1)  The  model  should  handle  uncertainties,  and  risks  should 
be  clearly  demonstrated  to  the  decision-maker. 

(2)  The  model  should  maintain  threshold  for  critical  factors 

(3)  The  procedure  (model)  should  be  accepted  by  the  decision 
maker  and  the  acceptance  cleared  at  an  early  stage. 
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(4)  Decision-makers  should  be  consulted  on  the  procedure 
structuring  at  its  initial  phases. 

(5)  Simplicity  is  a  paramount  feature  of  such  procedure 
or  model. 

In  general  it  must  be  said  that  the  MAUT  actual  implementation 
in  the  acquisition  decision-making  warrants  a  more  thorough 
discussion  than  allowed  by  the  questionnaire,  or  suggested  by 
the  answers  and  the  written  comments.  This  is  done  in  detail 
in  the  body  of  this  thesis. 
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Table  A-VII-10 


Statements  Selection  by  the  Israeli  Respondents 
(Selection  is  identical  to  agreement  with  the  statement) 


No .  Corresponds 
to  "American” 
_ Question  # 


The  Statanent 


Selected 

by 

(Amount) 


Selected 

by 

(Percent) 


1  Q28 

2  Q13 

3  Q14 

4  Q12 

5  Q33 

6  Q32 

7  Qll 


The  model  is  too  elaborate. 

Scores  only  are  sufficient. 

The  rest  should  be  left  to 
pure  judgment.  1 

The  model  is  supposed  to  be 
objective  therefore  may  not 
be  changed.  (The  decision¬ 
maker  is  not  bound  by  the 
model  outcomes.)  4 

Scores  and  weight  would  be 
determined  in  the  process 
prior  to  the  decision.  The 
decision-maker  nay  change 
them  according  to  his  own 
judgment.  7 

The  decision-maker  himself 

should  determine  weights 

and  scores.  2 

Weights  and  scores  should 
be  determined  by  independent 
evaluations  of  people  involved 
in  the  decision.  4 

Weights  and  scores  should 
be  determined  by  a  group 
of  experts.  7 

Such  a  model  should  be 
supplsnented  by  the  detailed 
data  and  underlying 
rationale.  6 


9 


36 


64 


18 


36 


64 


55 
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A- VI I I .  Relative  Importance  of  Value-Dimension  Categories 


A.  PURPOSE 

The  purpose  of  this  section  is  to  examine  whether  or  not 
some  systematic  characteristics  are  observed  in  the  relative 
importance  weight  assigned  to  three  value-dimension  categories 
(Direct,  Socio-economic,  Political)  by  the  questionnaire's 
respondents . 

This  examination  should  assist  in  defining  initial  steps 
in  MAUT  procedure  structuring  for  top-level  acquisition  deci¬ 
sion-making,  as  introduced  in  the  body  of  this  thesis. 

B .  METHODOLOGY 

The  analysis  of  the  above  mentioned  subject  is  based  on 
the  data  collected  in  Part  II  of  the  questionnaire.  In  Part 
II  the  respondents  were  asked  to  evaluate  the  relative  impor¬ 
tance  of  three  main  value  dimension  categories — Direct,  i.e., 
factors  that  affect  directly  the  military  forces.  Socio-economic, 
i.e.,  factors  that  have  an  impact  on  the  national  social  and 
economic  areas,  and  political,  i.e.,  the  external  and  internal 
political  aspects  of  the  decision.  This  evaluation  is  done  for 
several  decision  levels^  separately.  Each  respondent  was  asked 
to  refer  to  two  cases.  In  fact,  some  respondents  did  not  fill 
in  this  part  at  all.  Some  referred  to  one  case  only,  and  some 


With  this  respect,  DSARC  and  (S)SARC  have  been  considered 
as  decision  bodies  although  formally  they  have  recommending 
authority  only.  The  underlying  assumption  was  that  in  consoli¬ 
dating  their  recommendations  these  bodies  should  reach,  in  a 
sense,  some  sort  of  decision. 
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referred  only  to  those  levels  in  which  they  felt  sufficiently 
knowledgeable  to  make  the  evaluation.  Probably  in  quite  a 
few  cases  the  respondent  referred  to  the  same  system  (they 
were  not  asked  to  note  the  specific  system's  name).  To  examine 
what  bias  this  might  cause,  two  cases,  in  which  system  identi¬ 
fication  has  been  noted  by  some  of  the  respondents,  were 
checked--one  American  case  and  one  Israeli  case.  In  both 
cases,  individuals  that  referred  to  the  same  weapon  system, 
evaluated  the  relative  importance  of  the  categories  completely 
differently  from  each  other. 

For  each  column,  i.e.,  for  each  category  at  each  level  a 
simple  average  of  weights  through  all  cases  (systems)  is 
calculated.  The  absolute  numerical  value  of  these  averages, 
which  is  exhibited  in  Tables  A-VIII-1  and  A-VIII-2  has  no 
practical  meaning  since  it  can't  be  used  "as  is"  in  a  MAUT 
procedure  evaluation  for  any  specific  system.  This  is  obvious 
because  any  specific  decision  has  its  'specific'  weighting 
and  no  "general  (average)  weighting"  can  replace  it  (unless 
they  are  coincidentally  identical) .  But  the  relative  dimen¬ 
sions  of  the  averages  do  reveal  some  interesting  relations 
between  categories  and  between  levels. 

C.  CATEGORIES  WEIGHTING  CHARACTERISTICS 

Although  the  absolute  values  of  the  weights  in  a  corres¬ 
ponding  category  and  level  are  to  a  great  extent  divergent 
(e.g.,  the  weight  of  "direct"  category  at  SecDef  level  for  one 
system  might  be  25%  and  for  another — 90%) ,  there  are  some 
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Table  A-VIII- 
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Table  A-VIII-2 


Average  Importance  Weights  (Expressed  in  percentage) , 
Assigned  to  Three  Categories  (Direct,  Socio-Economic, 
Political)  In  Three  Decision  Levels,  as  Observed 
in  16  Cases  by  the  Israeli  Respondents 


Min.  of  Def .  General  HQs  IAF  HQs 


Dir 

SE 

Pol 

Dir 

SE 

Pol 

Dir 

SE 

Pol 

Average 

33.7 

36 

29.7 

53.6 

38.2 

8.2 

66.5 

22 

11.5 

Standard 

23.2 

16.4 

29.9 

31.4 

33.4 

12.5 

29.3 

27.6 

17.1 

Deviation 
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"rules"  (or  characteristics)  within  the  weighting  of  each 
single  system,  which  are  common  to  almost  100%  of  the  cases 
in  the  sample.  Thus  the  average  weights  also  relate  to  each 
other  according  to  these  same  rules,  which  are  surprisingly 
corresponding  to  what  was  expected  by  pure  common  sense.  The 
categories  weighting  characteristics  of  the  American  community 
can  be  summarized  as  follows: 

(1)  In  general,  the  higher  the  decision  level  is,  the  greater 
is  the  combined  weight  of  the  political  and  socio-economic 
categories,  relative  to  the  direct  category. 

(2)  The  recommending  bodies,  i.e.,  the  DSARC  and  (S)SARC 
tend  to  be  more  "direct"  category-oriented  than  their  decision¬ 
making  superiors,  i.e.,  SecDef  and  (S)Sec,  respectively.  (In 
some  cases  they  are  equally  "direct”  category-oriented  but 
almost  never  less  than  their  superiors. 

(3)  For  the  American  decision-makers  the  socio-economic 
category  weights  less  than  the  political  category,  and  its 
relative  weight  values  are  very  low  (10%  and  less)  for  almost 
all  cases  considered,  as  well  as  on  the  average. 

The  categories  weighting  characteristics  for  the  Israeli 
community  can  be  summarized  as  follows: 

(1)  Like  in  the  American  community,  the  higher  the  decision 
level  is,  the  greater  is  the  combined  weight  of  socio-economic 
and  political  categories  relative  to  the  direct  category. 

(2)  Different  from  the  American  community,  the  socio-economic 
category  tends  to  weigh  more  than  the  political  one. 


(3)  Through  all  levels  the  relative  weight  of  the  "direct" 
category  is  smaller,  and  that  of  the  combined  socio-economic 
and  political  categories  is  bigger,  than  for  the  corresponding 
levels  in  the  American  community. 

A  general  rule  is  that  emphasis  on  specific  category  is 
kept  through  all  levels.  For  example,  if  a  system  has  unusually 
high  political  weight  of  75%  at  SecDef  level,  it  has  also  an 
unusually  high  weight  of  60%  at  the  DSARC,  70%  at  (S)Sec, 
and  60%  at  (S)SARC  levels. 

D.  CHAPTER  SUMMARY 

The  categories  weighting  analysis  reveals  several  rules 
of  behavior,  as  discussed  in  this  chapter.  This  may  have 
an  impact  on  the  MAUT  value  tree  structuring  and  on  reducing 
the  amount  of  sensitivity  analysis  required.  This  is  discussed 
in  greater  detail  in  the  body  of  this  thesis. 


A-IX.  Identifying  Value-Dimensions 


A.  PURPOSE 

The  purpose  of  this  chapter  is  to  attempt  the  structuring  of 
an  initial  'value  tree’,  compatible  for  some  acquisition  "general 
cases",  by  identifying  its  value  dimensions  (or  in  other  words — 
attributes,  or  factors). 

B .  METHODOLOGY 

Part  VI  of  the  questionnaire  suggested  a  list  of  19  value 
dimensions.  The  respondents  were  asked  to  rate  each  dimension 
by  a  score  on  a  0-4  scale,  according  to  its  impact  on  a 
specific  system  decision.  The  purpose  for  the  rating  was 
to  identify  such  dimensions  that  have  repeating  0  or  1  scores, 
in  order  to  consider  their  elimination  from  the  list.  The 
possibility  of  finding  relative  weights  according  to  the  attri¬ 
butes  rating  was  abandoned  since  the  rating  dealt  with  differ¬ 
ent  systems  and  different  life  cycle  phases,  and  any  aggregate 
rating  weight  would  have  been  meaningless. 

Another  purpose  of  the  attributes  list  question  was  to 
stimulate  comments  about  the  various  attributes.  Finally, 
the  respondents  were  asked  to  suggest  additional  attributes 
which  they  thought  warranted  inclusion  in  the  list. 

C.  CONCLUSIONS 

The  value  dimensions  rating  did  not  identify  any  dimension 
that  should  be  eliminated.  Except  for  several  question  marks, 
there  were  no  written  comments  on  the  listed  attributes.  This 
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does  not  mean  that  the  list  can  be  structured  as  is  into  a 
MAUT  value  tree.  It  requires  quite  a  few  changes  and  adjust¬ 
ments,  as  discussed  in  the  body  of  this  thesis. 

Finally,  Part  VI  of  the  questionnaire  provided  a  list  of 
additional  value  dimensions,  as  suggested  by  the  respondents. 
The  "raw"  list  is  as  follows:^" 

(1)  FMS  potential 

(2)  Operating  flexibility  (e.g.,  V/STOL,  carrier 
suitability) 

(3)  Testing  results 

(4)  Affordability 

(5)  Producibility  (production  base,  production  readiness) 

(6)  Competition  (Second  Source) 

(7)  Mobilization  capability 

(8)  Complexity,  required  skill  levels 

(9)  Force  multiplier 

(10)  Growth  potential 

(11)  Contract  financing 

(12)  Contractual  terms/conditions 

(13)  Operational  suitability 

(14)  Cost  uncertainty 

(15)  Innovation  degree  (a  new  system,  an  improved  system, 
new  features) 

(16)  Meeting  threat  over  long  run 

(17)  Contribution  to  general  technological  advance 

^The  list  includes  the  additional  value  dimension  only, 
as  defined  by  the  respondents. 
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